Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X.XX.X.XXXX.X.X....X.....XXX.X.......X..X...................] duration: 4.284931081s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:19:26 UTC) orbiter warning: context deadline exceeded Duration: 4.28495203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X] duration: 30.002858559s trace: https://jaeger.dev.storj.io/trace/61a5622f546cc0a5 FAIL (2025-12-28 17:19:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: UKFJMSMCC4IUMQBVBQJXVSYB5S2JTXFBPPQQZI5CJCS3RKERCFFA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OI4BSOHB5T3RG7RSMG45SPD3BBE33NQV5V54KY6PYZDFFUJKKR5A) context deadline exceeded; piecestore: (Node ID: 17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, Piece ID: T4JH6MNERQJHD5NKMQSCQW27QK3KYU5JXHM3ASY5DOF2X5GJV7LA) context deadline exceeded ======LOGS====== Dec 28 17:18:39.450778120 (553h49m39.321861165s) [upload=485567]: Starting upload Dec 28 17:18:39.450846170 (553h49m39.321929215s) [upload=485567]: Done starting upload Dec 28 17:18:39.450904781 (553h49m39.321987816s) [upload=485567]: Uploading segments... Dec 28 17:18:39.451111735 (553h49m39.32219477s) [upload=485567]: Got next segment. Inline: false Dec 28 17:18:39.451142233 (553h49m39.322225268s) [upload=485567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:18:39.451147303 (553h49m39.322230338s) [upload=485567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:18:39.695021343 (553h49m39.566104388s) [seg_pos={0 0}, upload=485567]: Begin upload segment... Dec 28 17:18:39.695086047 (553h49m39.566169082s) [seg_pos={0 0}, upload=485567]: Done begin upload segment. Dec 28 17:18:39.695103350 (553h49m39.566186395s) [noise=true , address=45.91.116.8:4045 , piece_id=UKFJMSMCC4IUMQBVBQJXVSYB5S2JTXFBPPQQZI5CJCS3RKERCFFA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.695128567 (553h49m39.566211612s) [noise=true , address=45.91.116.4:4940 , piece_id=6SAD2AQKNMWQJOLSEYWYBVP6JFSPT5ZRKBUJPHVPUQ5UWYORUEEA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.695608161 (553h49m39.566691206s) [noise=true , address=145.79.227.11:4732 , piece_id=HMUVHUMDCYLZYGWPZJLEGR4OG6TLHTSEN7TKTDRUYCQ72ZD23DEA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.696270512 (553h49m39.567353547s) [noise=true , address=45.91.116.14:4191 , piece_id=4VOU7ZBXFRTKLEOSIQFLPZF4Z2TWVCZD5MZXHNPEKV6ZDOH77NRA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.696591743 (553h49m39.567674779s) [noise=true , address=145.79.227.38:4978 , piece_id=W34O2O72NQCPQKCBYXWOP7NQA7SVZC5ZV565MDDN5UFUA6DJ3XUQ, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.696874311 (553h49m39.567957356s) [noise=true , address=145.79.227.39:4015 , piece_id=4KMY7T7PBZ4FZSXUXUI3NAU7TASURUFXM6K32ILTUC2WV2QWE5DA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.697198489 (553h49m39.568281524s) [noise=true , address=77.90.31.21:4880 , piece_id=DOOVR6JTIX3VU3D6BLHU67YEG3KKR5FD4BN2S7EQ5ND4PEFMMNOA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.697565728 (553h49m39.568648763s) [noise=true , address=145.79.227.3:4151 , piece_id=CX2OWW7TKHHM4HG4E5CRTHHECTHPEE43HLHZE75H4GFUQECMHNFA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.698125464 (553h49m39.569208499s) [noise=true , address=145.79.227.16:4910 , piece_id=ARKTVOSSBTXWIO4ZCTJAEZTO3BB6MN42EUBV4SMXOIIMA5VVM3EQ, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.698139461 (553h49m39.569222506s) [noise=true , address=145.79.227.7:4295 , piece_id=3VJK767IJ2BYGQIWDTRXBGQSHYDSPNC2GDT2OHAGE5CTV62WK5MA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.698618564 (553h49m39.569701609s) [noise=true , address=45.91.116.11:4119 , piece_id=OI4BSOHB5T3RG7RSMG45SPD3BBE33NQV5V54KY6PYZDFFUJKKR5A, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.699067077 (553h49m39.570150122s) [noise=true , address=45.91.116.6:4973 , piece_id=GG7DIXDLHF2HUPCR4CTVC2E6IPSJWOODAQ4MYUM2VQIGP762PLLA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.699648144 (553h49m39.570731179s) [noise=true , address=145.79.227.10:4401 , piece_id=TQ7OROXJK3E2YG2LHYIULLKUHN4XRO76HLNHUZE52Q54DMNRRABA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.700124902 (553h49m39.571207937s) [noise=true , address=145.79.227.36:4518 , piece_id=T4JH6MNERQJHD5NKMQSCQW27QK3KYU5JXHM3ASY5DOF2X5GJV7LA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.701108334 (553h49m39.572191379s) [noise=true , address=145.79.227.14:4840 , piece_id=JUHEWQXYUOXJRL6TJGATFY7HEAHMYOGVADAEJAQ6Y5HEINBMSULA, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:39.701583690 (553h49m39.572666725s) [noise=true , address=45.91.116.13:4155 , piece_id=S6UXMHMG2FABPKHPRPIU6AHLJ37HX455C3Z55FN3NMLAX4I2SCGQ, seg_pos={0 0}, upload=485567]: Uploading piece... Dec 28 17:18:41.408299021 (553h49m41.279382056s) [upload=485567]: Next returned nil segment Dec 28 17:18:41.408320311 (553h49m41.279403356s) [upload=485567]: Waiting for error group managing segments... Dec 28 17:18:41.700673305 (553h49m41.57175635s) [noise=true , address=145.79.227.10:4401 , piece_id=TQ7OROXJK3E2YG2LHYIULLKUHN4XRO76HLNHUZE52Q54DMNRRABA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.705165168 (553h49m41.576248203s) [noise=true , address=145.79.227.7:4295 , piece_id=3VJK767IJ2BYGQIWDTRXBGQSHYDSPNC2GDT2OHAGE5CTV62WK5MA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.713485882 (553h49m41.584568927s) [noise=true , address=145.79.227.14:4840 , piece_id=JUHEWQXYUOXJRL6TJGATFY7HEAHMYOGVADAEJAQ6Y5HEINBMSULA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.713698497 (553h49m41.584781542s) [noise=true , address=145.79.227.11:4732 , piece_id=HMUVHUMDCYLZYGWPZJLEGR4OG6TLHTSEN7TKTDRUYCQ72ZD23DEA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.715291871 (553h49m41.586374916s) [noise=true , address=145.79.227.16:4910 , piece_id=ARKTVOSSBTXWIO4ZCTJAEZTO3BB6MN42EUBV4SMXOIIMA5VVM3EQ, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.718808698 (553h49m41.589891733s) [noise=true , address=145.79.227.39:4015 , piece_id=4KMY7T7PBZ4FZSXUXUI3NAU7TASURUFXM6K32ILTUC2WV2QWE5DA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.731602059 (553h49m41.602685094s) [noise=true , address=145.79.227.3:4151 , piece_id=CX2OWW7TKHHM4HG4E5CRTHHECTHPEE43HLHZE75H4GFUQECMHNFA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.731778685 (553h49m41.60286176s) [noise=true , address=145.79.227.36:4518 , piece_id=T4JH6MNERQJHD5NKMQSCQW27QK3KYU5JXHM3ASY5DOF2X5GJV7LA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.796619484 (553h49m41.667702529s) [noise=true , address=145.79.227.38:4978 , piece_id=W34O2O72NQCPQKCBYXWOP7NQA7SVZC5ZV565MDDN5UFUA6DJ3XUQ, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.928589660 (553h49m41.799672705s) [noise=true , address=77.90.31.21:4880 , piece_id=DOOVR6JTIX3VU3D6BLHU67YEG3KKR5FD4BN2S7EQ5ND4PEFMMNOA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:41.972774977 (553h49m41.843858022s) [noise=true , address=45.91.116.8:4045 , piece_id=UKFJMSMCC4IUMQBVBQJXVSYB5S2JTXFBPPQQZI5CJCS3RKERCFFA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:42.026774352 (553h49m41.897857397s) [noise=true , address=45.91.116.11:4119 , piece_id=OI4BSOHB5T3RG7RSMG45SPD3BBE33NQV5V54KY6PYZDFFUJKKR5A, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:42.239782288 (553h49m42.110865333s) [noise=true , address=45.91.116.6:4973 , piece_id=GG7DIXDLHF2HUPCR4CTVC2E6IPSJWOODAQ4MYUM2VQIGP762PLLA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: <nil> Dec 28 17:18:42.239806424 (553h49m42.110889459s) [seg_pos={0 0}, upload=485567]: Segment reached optimal threshold of 13 pieces. Dec 28 17:18:42.240028807 (553h49m42.111111842s) [noise=true , address=45.91.116.14:4191 , piece_id=4VOU7ZBXFRTKLEOSIQFLPZF4Z2TWVCZD5MZXHNPEKV6ZDOH77NRA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 28 17:18:42.240038525 (553h49m42.11112156s) [noise=true , address=45.91.116.14:4191 , piece_id=4VOU7ZBXFRTKLEOSIQFLPZF4Z2TWVCZD5MZXHNPEKV6ZDOH77NRA, seg_pos={0 0}, upload=485567]: Long tail cancellation due to optimal threshold Dec 28 17:18:42.240198811 (553h49m42.111281846s) [noise=true , address=45.91.116.13:4155 , piece_id=S6UXMHMG2FABPKHPRPIU6AHLJ37HX455C3Z55FN3NMLAX4I2SCGQ, seg_pos={0 0}, upload=485567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 17:18:42.240206976 (553h49m42.111290011s) [noise=true , address=45.91.116.13:4155 , piece_id=S6UXMHMG2FABPKHPRPIU6AHLJ37HX455C3Z55FN3NMLAX4I2SCGQ, seg_pos={0 0}, upload=485567]: Long tail cancellation due to optimal threshold Dec 28 17:18:42.240395385 (553h49m42.111478451s) [noise=true , address=45.91.116.4:4940 , piece_id=6SAD2AQKNMWQJOLSEYWYBVP6JFSPT5ZRKBUJPHVPUQ5UWYORUEEA, seg_pos={0 0}, upload=485567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 17:18:42.240415473 (553h49m42.111498528s) [noise=true , address=45.91.116.4:4940 , piece_id=6SAD2AQKNMWQJOLSEYWYBVP6JFSPT5ZRKBUJPHVPUQ5UWYORUEEA, seg_pos={0 0}, upload=485567]: Long tail cancellation due to optimal threshold Dec 28 17:18:42.240445871 (553h49m42.111528906s) [seg_pos={0 0}, upload=485567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:18:42.240557664 (553h49m42.111640719s) [upload=485567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:18:42.240565068 (553h49m42.111648103s) [upload=485567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:18:42.532095688 (553h49m42.403178763s) [upload=485567]: Done uploading segments... Dec 28 17:18:42.532123610 (553h49m42.403206655s) [upload=485567]: Upload finished. err: <nil> Duration: 30.002882403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XX.XX.X.XXXX.X.X....X.....XXX.X.......X..X..................] duration: 3.482593541s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:18:58 UTC) orbiter warning: context deadline exceeded Duration: 3.482621584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.] duration: 6.188282069s trace: https://jaeger.dev.storj.io/trace/52c470ea5b9cce7 FAIL (2025-12-28 17:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: H2QV5SOUODTRVODDVWM65XTN7YCTTHH2I7HWG6R6CBGTIGV67KYQ) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCAQRVGOLK6IAISHBWOKLDC4KEHDLHLZKPWJSDUJ7SX6F5UGTPYQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: P54LW3GEI3CO56VVVQ6Q7CJ2RECQAMRX6NHZP6HOPANI4LDJPQBQ) context deadline exceeded ======LOGS====== 8 17:17:41.884112407 (553h48m41.755195452s) [upload=485557]: Starting upload Dec 28 17:17:41.884196156 (553h48m41.755279211s) [upload=485557]: Done starting upload Dec 28 17:17:41.884342716 (553h48m41.755425751s) [upload=485557]: Uploading segments... Dec 28 17:17:41.884494946 (553h48m41.755578001s) [upload=485557]: Got next segment. Inline: false Dec 28 17:17:41.884554570 (553h48m41.755637615s) [upload=485557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:17:41.884561182 (553h48m41.755644217s) [upload=485557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:17:42.128101698 (553h48m41.999184743s) [seg_pos={0 0}, upload=485557]: Begin upload segment... Dec 28 17:17:42.128155341 (553h48m41.999238376s) [seg_pos={0 0}, upload=485557]: Done begin upload segment. Dec 28 17:17:42.128190648 (553h48m41.999273683s) [noise=true , address=77.90.31.17:4773 , piece_id=H2QV5SOUODTRVODDVWM65XTN7YCTTHH2I7HWG6R6CBGTIGV67KYQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.128226426 (553h48m41.999309471s) [noise=true , address=45.91.116.12:4228 , piece_id=LCAQRVGOLK6IAISHBWOKLDC4KEHDLHLZKPWJSDUJ7SX6F5UGTPYQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.128859311 (553h48m41.999942356s) [noise=true , address=145.79.227.46:4602 , piece_id=PPR4PHHMVBFOE7LYUMCZLFXGTDJ7SMLDKYH7ILH3FR2A4IHKWGBA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.129422393 (553h48m42.000505438s) [noise=true , address=45.91.116.11:4118 , piece_id=RRJV4DOYR57FBXTQH327BK5J5NYJPXPYBC3HJFQSNNJKY2USTOKA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.130231363 (553h48m42.001314398s) [noise=true , address=145.79.227.51:4410 , piece_id=VALAWJL65JZXOEMWAPDTIMVCQL4BQPOJUVRNUBGFM5DAWN63VKUA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.130683795 (553h48m42.00176683s) [noise=true , address=145.79.227.47:4267 , piece_id=KDRLOG32E6H5LIRACC5ULPLFX55MN3PL2ROZTZF76GVQLYD6H26Q, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.130694626 (553h48m42.001777671s) [noise=true , address=145.79.227.3:4149 , piece_id=XZSBXQQ4ZMI6SRLYTTO6FG766AYFPGDP4FGNPBT5Q2VKRHOK7WOQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.131039423 (553h48m42.002122458s) [noise=true , address=145.79.227.14:4837 , piece_id=XVAPHD56GG2JEO5TZHBOKBI3KXXV3ETYWBJFH5Z5KA6X64UZZVQA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.131389709 (553h48m42.002472765s) [noise=true , address=145.79.227.2:4113 , piece_id=KXRVHVU5ISMYRC5K7HRDXVAO6VSV2FXKF2WDRQ4A7XEQZ7QUNPEQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.131454202 (553h48m42.002537237s) [noise=true , address=145.79.227.15:4875 , piece_id=KCMGFKMRFXS7G2Y7PEQSLXXP5UZ45UNGVBHNFSZL47WZUF7INMEA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.131789811 (553h48m42.002872846s) [noise=true , address=145.79.227.38:4977 , piece_id=JKKSP5UZXGAVCQFNBADABK3UNYK2ZFQQGZSQ42M7RYQMKMTFBXIA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.132048063 (553h48m42.003131118s) [noise=true , address=45.91.116.9:4262 , piece_id=P54LW3GEI3CO56VVVQ6Q7CJ2RECQAMRX6NHZP6HOPANI4LDJPQBQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.132103057 (553h48m42.003186092s) [noise=true , address=145.79.227.42:4122 , piece_id=FGD3K5WKP3ML77KOSJKDPJIW2ZIQA3W7TAY475VF3XNSKTOZO4DA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.132745321 (553h48m42.003828366s) [noise=true , address=45.91.116.6:4973 , piece_id=YYCC7XLIA2Y6BJMPSGSY27Y7USEXK6ERCFZWV7XDKJWP7HNHSR6A, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.132790166 (553h48m42.003873211s) [noise=true , address=77.90.31.19:4844 , piece_id=LIHOZYZ4MB25G6WOJHFFMBZ3HYL54SH5WCYNTEEJOJMBWJFEHW4A, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.133467215 (553h48m42.00455026s) [noise=true , address=145.79.227.50:4373 , piece_id=R6A7QSMZMJFCDAIE66BISCK3Z2IF5PPRSVDWW7G6HXJTJJTNXHIA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:43.850760077 (553h48m43.721843113s) [upload=485557]: Next returned nil segment Dec 28 17:17:43.850777159 (553h48m43.721860215s) [upload=485557]: Waiting for error group managing segments... Dec 28 17:17:44.138706831 (553h48m44.009789876s) [noise=true , address=145.79.227.38:4977 , piece_id=JKKSP5UZXGAVCQFNBADABK3UNYK2ZFQQGZSQ42M7RYQMKMTFBXIA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.139384672 (553h48m44.010467717s) [noise=true , address=145.79.227.15:4875 , piece_id=KCMGFKMRFXS7G2Y7PEQSLXXP5UZ45UNGVBHNFSZL47WZUF7INMEA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.140140791 (553h48m44.011223836s) [noise=true , address=145.79.227.2:4113 , piece_id=KXRVHVU5ISMYRC5K7HRDXVAO6VSV2FXKF2WDRQ4A7XEQZ7QUNPEQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.145361232 (553h48m44.016444267s) [noise=true , address=145.79.227.47:4267 , piece_id=KDRLOG32E6H5LIRACC5ULPLFX55MN3PL2ROZTZF76GVQLYD6H26Q, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.152414824 (553h48m44.023497859s) [noise=true , address=145.79.227.42:4122 , piece_id=FGD3K5WKP3ML77KOSJKDPJIW2ZIQA3W7TAY475VF3XNSKTOZO4DA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.153209958 (553h48m44.024293003s) [noise=true , address=145.79.227.3:4149 , piece_id=XZSBXQQ4ZMI6SRLYTTO6FG766AYFPGDP4FGNPBT5Q2VKRHOK7WOQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.153988680 (553h48m44.025071715s) [noise=true , address=145.79.227.46:4602 , piece_id=PPR4PHHMVBFOE7LYUMCZLFXGTDJ7SMLDKYH7ILH3FR2A4IHKWGBA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.155450193 (553h48m44.026533228s) [noise=true , address=145.79.227.51:4410 , piece_id=VALAWJL65JZXOEMWAPDTIMVCQL4BQPOJUVRNUBGFM5DAWN63VKUA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.156411604 (553h48m44.027494639s) [noise=true , address=145.79.227.50:4373 , piece_id=R6A7QSMZMJFCDAIE66BISCK3Z2IF5PPRSVDWW7G6HXJTJJTNXHIA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.161288030 (553h48m44.032371075s) [noise=true , address=145.79.227.14:4837 , piece_id=XVAPHD56GG2JEO5TZHBOKBI3KXXV3ETYWBJFH5Z5KA6X64UZZVQA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.375386914 (553h48m44.246469949s) [noise=true , address=45.91.116.12:4228 , piece_id=LCAQRVGOLK6IAISHBWOKLDC4KEHDLHLZKPWJSDUJ7SX6F5UGTPYQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.430003254 (553h48m44.301086299s) [noise=true , address=45.91.116.9:4262 , piece_id=P54LW3GEI3CO56VVVQ6Q7CJ2RECQAMRX6NHZP6HOPANI4LDJPQBQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.436999386 (553h48m44.308082431s) [noise=true , address=77.90.31.17:4773 , piece_id=H2QV5SOUODTRVODDVWM65XTN7YCTTHH2I7HWG6R6CBGTIGV67KYQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.437022619 (553h48m44.308105654s) [seg_pos={0 0}, upload=485557]: Segment reached optimal threshold of 13 pieces. Dec 28 17:17:44.437286101 (553h48m44.308369146s) [noise=true , address=45.91.116.11:4118 , piece_id=RRJV4DOYR57FBXTQH327BK5J5NYJPXPYBC3HJFQSNNJKY2USTOKA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 17:17:44.437296230 (553h48m44.308379275s) [noise=true , address=45.91.116.11:4118 , piece_id=RRJV4DOYR57FBXTQH327BK5J5NYJPXPYBC3HJFQSNNJKY2USTOKA, seg_pos={0 0}, upload=485557]: Long tail cancellation due to optimal threshold Dec 28 17:17:44.437413122 (553h48m44.308496158s) [noise=true , address=45.91.116.6:4973 , piece_id=YYCC7XLIA2Y6BJMPSGSY27Y7USEXK6ERCFZWV7XDKJWP7HNHSR6A, seg_pos={0 0}, upload=485557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 17:17:44.437442949 (553h48m44.308525984s) [noise=true , address=45.91.116.6:4973 , piece_id=YYCC7XLIA2Y6BJMPSGSY27Y7USEXK6ERCFZWV7XDKJWP7HNHSR6A, seg_pos={0 0}, upload=485557]: Long tail cancellation due to optimal threshold Dec 28 17:17:44.437611641 (553h48m44.308694676s) [noise=true , address=77.90.31.19:4844 , piece_id=LIHOZYZ4MB25G6WOJHFFMBZ3HYL54SH5WCYNTEEJOJMBWJFEHW4A, seg_pos={0 0}, upload=485557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:46286->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:46286->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46286->77.90.31.19:4844: use of closed network connection Dec 28 17:17:44.437619295 (553h48m44.30870233s) [noise=true , address=77.90.31.19:4844 , piece_id=LIHOZYZ4MB25G6WOJHFFMBZ3HYL54SH5WCYNTEEJOJMBWJFEHW4A, seg_pos={0 0}, upload=485557]: Long tail cancellation due to optimal threshold Dec 28 17:17:44.437637420 (553h48m44.308720465s) [seg_pos={0 0}, upload=485557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:17:44.437738122 (553h48m44.308821167s) [upload=485557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:17:44.437744564 (553h48m44.308827609s) [upload=485557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:17:44.722122803 (553h48m44.593205848s) [upload=485557]: Done uploading segments... Dec 28 17:17:44.722144514 (553h48m44.593227599s) [upload=485557]: Upload finished. err: <nil> Duration: 6.188296496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X........................X.......................] duration: 44.075808491s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 44.075823831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X] duration: 30.001851964s trace: https://jaeger.dev.storj.io/trace/52c470ea5b9cce7 FAIL (2025-12-28 17:18:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: H2QV5SOUODTRVODDVWM65XTN7YCTTHH2I7HWG6R6CBGTIGV67KYQ) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCAQRVGOLK6IAISHBWOKLDC4KEHDLHLZKPWJSDUJ7SX6F5UGTPYQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: P54LW3GEI3CO56VVVQ6Q7CJ2RECQAMRX6NHZP6HOPANI4LDJPQBQ) context deadline exceeded ======LOGS====== 8 17:17:41.884112407 (553h48m41.755195452s) [upload=485557]: Starting upload Dec 28 17:17:41.884196156 (553h48m41.755279211s) [upload=485557]: Done starting upload Dec 28 17:17:41.884342716 (553h48m41.755425751s) [upload=485557]: Uploading segments... Dec 28 17:17:41.884494946 (553h48m41.755578001s) [upload=485557]: Got next segment. Inline: false Dec 28 17:17:41.884554570 (553h48m41.755637615s) [upload=485557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:17:41.884561182 (553h48m41.755644217s) [upload=485557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:17:42.128101698 (553h48m41.999184743s) [seg_pos={0 0}, upload=485557]: Begin upload segment... Dec 28 17:17:42.128155341 (553h48m41.999238376s) [seg_pos={0 0}, upload=485557]: Done begin upload segment. Dec 28 17:17:42.128190648 (553h48m41.999273683s) [noise=true , address=77.90.31.17:4773 , piece_id=H2QV5SOUODTRVODDVWM65XTN7YCTTHH2I7HWG6R6CBGTIGV67KYQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.128226426 (553h48m41.999309471s) [noise=true , address=45.91.116.12:4228 , piece_id=LCAQRVGOLK6IAISHBWOKLDC4KEHDLHLZKPWJSDUJ7SX6F5UGTPYQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.128859311 (553h48m41.999942356s) [noise=true , address=145.79.227.46:4602 , piece_id=PPR4PHHMVBFOE7LYUMCZLFXGTDJ7SMLDKYH7ILH3FR2A4IHKWGBA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.129422393 (553h48m42.000505438s) [noise=true , address=45.91.116.11:4118 , piece_id=RRJV4DOYR57FBXTQH327BK5J5NYJPXPYBC3HJFQSNNJKY2USTOKA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.130231363 (553h48m42.001314398s) [noise=true , address=145.79.227.51:4410 , piece_id=VALAWJL65JZXOEMWAPDTIMVCQL4BQPOJUVRNUBGFM5DAWN63VKUA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.130683795 (553h48m42.00176683s) [noise=true , address=145.79.227.47:4267 , piece_id=KDRLOG32E6H5LIRACC5ULPLFX55MN3PL2ROZTZF76GVQLYD6H26Q, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.130694626 (553h48m42.001777671s) [noise=true , address=145.79.227.3:4149 , piece_id=XZSBXQQ4ZMI6SRLYTTO6FG766AYFPGDP4FGNPBT5Q2VKRHOK7WOQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.131039423 (553h48m42.002122458s) [noise=true , address=145.79.227.14:4837 , piece_id=XVAPHD56GG2JEO5TZHBOKBI3KXXV3ETYWBJFH5Z5KA6X64UZZVQA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.131389709 (553h48m42.002472765s) [noise=true , address=145.79.227.2:4113 , piece_id=KXRVHVU5ISMYRC5K7HRDXVAO6VSV2FXKF2WDRQ4A7XEQZ7QUNPEQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.131454202 (553h48m42.002537237s) [noise=true , address=145.79.227.15:4875 , piece_id=KCMGFKMRFXS7G2Y7PEQSLXXP5UZ45UNGVBHNFSZL47WZUF7INMEA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.131789811 (553h48m42.002872846s) [noise=true , address=145.79.227.38:4977 , piece_id=JKKSP5UZXGAVCQFNBADABK3UNYK2ZFQQGZSQ42M7RYQMKMTFBXIA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.132048063 (553h48m42.003131118s) [noise=true , address=45.91.116.9:4262 , piece_id=P54LW3GEI3CO56VVVQ6Q7CJ2RECQAMRX6NHZP6HOPANI4LDJPQBQ, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.132103057 (553h48m42.003186092s) [noise=true , address=145.79.227.42:4122 , piece_id=FGD3K5WKP3ML77KOSJKDPJIW2ZIQA3W7TAY475VF3XNSKTOZO4DA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.132745321 (553h48m42.003828366s) [noise=true , address=45.91.116.6:4973 , piece_id=YYCC7XLIA2Y6BJMPSGSY27Y7USEXK6ERCFZWV7XDKJWP7HNHSR6A, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.132790166 (553h48m42.003873211s) [noise=true , address=77.90.31.19:4844 , piece_id=LIHOZYZ4MB25G6WOJHFFMBZ3HYL54SH5WCYNTEEJOJMBWJFEHW4A, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:42.133467215 (553h48m42.00455026s) [noise=true , address=145.79.227.50:4373 , piece_id=R6A7QSMZMJFCDAIE66BISCK3Z2IF5PPRSVDWW7G6HXJTJJTNXHIA, seg_pos={0 0}, upload=485557]: Uploading piece... Dec 28 17:17:43.850760077 (553h48m43.721843113s) [upload=485557]: Next returned nil segment Dec 28 17:17:43.850777159 (553h48m43.721860215s) [upload=485557]: Waiting for error group managing segments... Dec 28 17:17:44.138706831 (553h48m44.009789876s) [noise=true , address=145.79.227.38:4977 , piece_id=JKKSP5UZXGAVCQFNBADABK3UNYK2ZFQQGZSQ42M7RYQMKMTFBXIA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.139384672 (553h48m44.010467717s) [noise=true , address=145.79.227.15:4875 , piece_id=KCMGFKMRFXS7G2Y7PEQSLXXP5UZ45UNGVBHNFSZL47WZUF7INMEA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.140140791 (553h48m44.011223836s) [noise=true , address=145.79.227.2:4113 , piece_id=KXRVHVU5ISMYRC5K7HRDXVAO6VSV2FXKF2WDRQ4A7XEQZ7QUNPEQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.145361232 (553h48m44.016444267s) [noise=true , address=145.79.227.47:4267 , piece_id=KDRLOG32E6H5LIRACC5ULPLFX55MN3PL2ROZTZF76GVQLYD6H26Q, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.152414824 (553h48m44.023497859s) [noise=true , address=145.79.227.42:4122 , piece_id=FGD3K5WKP3ML77KOSJKDPJIW2ZIQA3W7TAY475VF3XNSKTOZO4DA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.153209958 (553h48m44.024293003s) [noise=true , address=145.79.227.3:4149 , piece_id=XZSBXQQ4ZMI6SRLYTTO6FG766AYFPGDP4FGNPBT5Q2VKRHOK7WOQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.153988680 (553h48m44.025071715s) [noise=true , address=145.79.227.46:4602 , piece_id=PPR4PHHMVBFOE7LYUMCZLFXGTDJ7SMLDKYH7ILH3FR2A4IHKWGBA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.155450193 (553h48m44.026533228s) [noise=true , address=145.79.227.51:4410 , piece_id=VALAWJL65JZXOEMWAPDTIMVCQL4BQPOJUVRNUBGFM5DAWN63VKUA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.156411604 (553h48m44.027494639s) [noise=true , address=145.79.227.50:4373 , piece_id=R6A7QSMZMJFCDAIE66BISCK3Z2IF5PPRSVDWW7G6HXJTJJTNXHIA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.161288030 (553h48m44.032371075s) [noise=true , address=145.79.227.14:4837 , piece_id=XVAPHD56GG2JEO5TZHBOKBI3KXXV3ETYWBJFH5Z5KA6X64UZZVQA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.375386914 (553h48m44.246469949s) [noise=true , address=45.91.116.12:4228 , piece_id=LCAQRVGOLK6IAISHBWOKLDC4KEHDLHLZKPWJSDUJ7SX6F5UGTPYQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.430003254 (553h48m44.301086299s) [noise=true , address=45.91.116.9:4262 , piece_id=P54LW3GEI3CO56VVVQ6Q7CJ2RECQAMRX6NHZP6HOPANI4LDJPQBQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.436999386 (553h48m44.308082431s) [noise=true , address=77.90.31.17:4773 , piece_id=H2QV5SOUODTRVODDVWM65XTN7YCTTHH2I7HWG6R6CBGTIGV67KYQ, seg_pos={0 0}, upload=485557]: Done uploading piece. err: <nil> Dec 28 17:17:44.437022619 (553h48m44.308105654s) [seg_pos={0 0}, upload=485557]: Segment reached optimal threshold of 13 pieces. Dec 28 17:17:44.437286101 (553h48m44.308369146s) [noise=true , address=45.91.116.11:4118 , piece_id=RRJV4DOYR57FBXTQH327BK5J5NYJPXPYBC3HJFQSNNJKY2USTOKA, seg_pos={0 0}, upload=485557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 17:17:44.437296230 (553h48m44.308379275s) [noise=true , address=45.91.116.11:4118 , piece_id=RRJV4DOYR57FBXTQH327BK5J5NYJPXPYBC3HJFQSNNJKY2USTOKA, seg_pos={0 0}, upload=485557]: Long tail cancellation due to optimal threshold Dec 28 17:17:44.437413122 (553h48m44.308496158s) [noise=true , address=45.91.116.6:4973 , piece_id=YYCC7XLIA2Y6BJMPSGSY27Y7USEXK6ERCFZWV7XDKJWP7HNHSR6A, seg_pos={0 0}, upload=485557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 17:17:44.437442949 (553h48m44.308525984s) [noise=true , address=45.91.116.6:4973 , piece_id=YYCC7XLIA2Y6BJMPSGSY27Y7USEXK6ERCFZWV7XDKJWP7HNHSR6A, seg_pos={0 0}, upload=485557]: Long tail cancellation due to optimal threshold Dec 28 17:17:44.437611641 (553h48m44.308694676s) [noise=true , address=77.90.31.19:4844 , piece_id=LIHOZYZ4MB25G6WOJHFFMBZ3HYL54SH5WCYNTEEJOJMBWJFEHW4A, seg_pos={0 0}, upload=485557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:46286->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:46286->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46286->77.90.31.19:4844: use of closed network connection Dec 28 17:17:44.437619295 (553h48m44.30870233s) [noise=true , address=77.90.31.19:4844 , piece_id=LIHOZYZ4MB25G6WOJHFFMBZ3HYL54SH5WCYNTEEJOJMBWJFEHW4A, seg_pos={0 0}, upload=485557]: Long tail cancellation due to optimal threshold Dec 28 17:17:44.437637420 (553h48m44.308720465s) [seg_pos={0 0}, upload=485557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:17:44.437738122 (553h48m44.308821167s) [upload=485557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:17:44.437744564 (553h48m44.308827609s) [upload=485557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:17:44.722122803 (553h48m44.593205848s) [upload=485557]: Done uploading segments... Dec 28 17:17:44.722144514 (553h48m44.593227599s) [upload=485557]: Upload finished. err: <nil> Duration: 30.001877032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X.................] duration: 3.645277972s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:18:06 UTC) orbiter warning: context deadline exceeded Duration: 3.645307328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X........................X......................] duration: 37.587730568s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:17:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 37.587745026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX...] duration: 5.707265147s trace: https://jaeger.dev.storj.io/trace/78b2fbe3857c51da FAIL (2025-12-28 17:17:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ) context deadline exceeded ======LOGS====== ec 28 17:14:47.122672129 (553h45m46.993755164s) [upload=485513]: Done starting upload Dec 28 17:14:47.122848355 (553h45m46.99393139s) [upload=485513]: Uploading segments... Dec 28 17:14:47.122940490 (553h45m46.994023525s) [upload=485513]: Got next segment. Inline: false Dec 28 17:14:47.122993632 (553h45m46.994076667s) [upload=485513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:14:47.123000024 (553h45m46.994083069s) [upload=485513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:14:47.366750740 (553h45m47.237833775s) [seg_pos={0 0}, upload=485513]: Begin upload segment... Dec 28 17:14:47.366798211 (553h45m47.237881256s) [seg_pos={0 0}, upload=485513]: Done begin upload segment. Dec 28 17:14:47.366833639 (553h45m47.237916674s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.366880908 (553h45m47.237963954s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.367791242 (553h45m47.238874287s) [noise=true , address=45.91.116.8:4046 , piece_id=NDZQO5FNABBKZVPB7WLPJKKXEFJCOW6WXFCSMQSU6G5O7ZOKJHWQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.367882796 (553h45m47.238965832s) [noise=true , address=45.91.116.13:4155 , piece_id=Y4R4FJGFL5UHC5GXSUL3HICW7D73VHCNYOYUI4NZZWVEKJYU3XDQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368120119 (553h45m47.239203154s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6KPTDQBX5XQWTNRJNN4LABJNMVSKSQGMXTEUKDXPL6BUWZLH5ZQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368655628 (553h45m47.239738663s) [noise=true , address=45.91.116.12:4225 , piece_id=ZX77FYRC46C2A7JO4BTKKMYL2RSDL3ME25SH4NS5TLDIK2Q6CINQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368879113 (553h45m47.239962158s) [noise=true , address=77.90.31.13:4629 , piece_id=2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.369518130 (553h45m47.240601165s) [noise=true , address=145.79.227.9:4366 , piece_id=MGYFRZHCXRP2LEN4DRFJJHQA6XMF5GCXUTEDFFDQOQ5LI4JC47CA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.369691581 (553h45m47.240774626s) [noise=true , address=145.79.227.8:4330 , piece_id=DRKFE3CPPP3CCMEQY63EQKYQXYU5PZPEJCIPCN3TO74LS7LZRT5A, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.370280673 (553h45m47.241363708s) [noise=true , address=145.79.227.15:4874 , piece_id=RDI2QQV23SKNFQT345S45FG35VQIUQWT3LYRAIOYLTNVDGHLMLLA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.370620850 (553h45m47.241703895s) [noise=true , address=145.79.227.14:4837 , piece_id=J2VEW6D6G7QQCBDSBNYUDWBQUTQY65J2SCO4UPNHDTX2B7XKSM3A, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371102617 (553h45m47.242185662s) [noise=true , address=145.79.227.32:4835 , piece_id=OYYIQY23ZCU2W3YU3MMF7F4YQJLD674YT5SMCHCS5RI5NCHR55FQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371447925 (553h45m47.24253097s) [noise=true , address=45.91.116.14:4191 , piece_id=HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371857875 (553h45m47.242940911s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.372547799 (553h45m47.243630834s) [noise=true , address=145.79.227.2:4115 , piece_id=OYPCRTHCLZPXDH4YRVZP3OMK7U4WIYUBCA4FCQ5TODI5COUZEJ7Q, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.373321012 (553h45m47.244404057s) [noise=true , address=145.79.227.16:4911 , piece_id=MCF3JNDKZR7C6OHM3YWN7RQQTKEYO657BJHPCMK34U44YK3E5ZVQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:49.084979743 (553h45m48.956062788s) [upload=485513]: Next returned nil segment Dec 28 17:14:49.084993810 (553h45m48.956076855s) [upload=485513]: Waiting for error group managing segments... Dec 28 17:14:49.370078022 (553h45m49.241161067s) [noise=true , address=145.79.227.15:4874 , piece_id=RDI2QQV23SKNFQT345S45FG35VQIUQWT3LYRAIOYLTNVDGHLMLLA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.370687373 (553h45m49.241770418s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6KPTDQBX5XQWTNRJNN4LABJNMVSKSQGMXTEUKDXPL6BUWZLH5ZQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.385089026 (553h45m49.256172072s) [noise=true , address=145.79.227.2:4115 , piece_id=OYPCRTHCLZPXDH4YRVZP3OMK7U4WIYUBCA4FCQ5TODI5COUZEJ7Q, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.387283836 (553h45m49.258366871s) [noise=true , address=145.79.227.9:4366 , piece_id=MGYFRZHCXRP2LEN4DRFJJHQA6XMF5GCXUTEDFFDQOQ5LI4JC47CA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.390730008 (553h45m49.261813053s) [noise=true , address=145.79.227.8:4330 , piece_id=DRKFE3CPPP3CCMEQY63EQKYQXYU5PZPEJCIPCN3TO74LS7LZRT5A, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.395841702 (553h45m49.266924737s) [noise=true , address=145.79.227.14:4837 , piece_id=J2VEW6D6G7QQCBDSBNYUDWBQUTQY65J2SCO4UPNHDTX2B7XKSM3A, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.402703177 (553h45m49.273786212s) [noise=true , address=145.79.227.16:4911 , piece_id=MCF3JNDKZR7C6OHM3YWN7RQQTKEYO657BJHPCMK34U44YK3E5ZVQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.408552185 (553h45m49.27963523s) [noise=true , address=145.79.227.32:4835 , piece_id=OYYIQY23ZCU2W3YU3MMF7F4YQJLD674YT5SMCHCS5RI5NCHR55FQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.588393496 (553h45m49.459476541s) [noise=true , address=45.91.116.14:4191 , piece_id=HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.615936172 (553h45m49.487019217s) [noise=true , address=77.90.31.13:4629 , piece_id=2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.751406332 (553h45m49.622489367s) [noise=true , address=45.91.116.8:4046 , piece_id=NDZQO5FNABBKZVPB7WLPJKKXEFJCOW6WXFCSMQSU6G5O7ZOKJHWQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.021029976 (553h45m49.892113011s) [noise=true , address=45.91.116.13:4155 , piece_id=Y4R4FJGFL5UHC5GXSUL3HICW7D73VHCNYOYUI4NZZWVEKJYU3XDQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.341406011 (553h45m50.212489056s) [noise=true , address=45.91.116.12:4225 , piece_id=ZX77FYRC46C2A7JO4BTKKMYL2RSDL3ME25SH4NS5TLDIK2Q6CINQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.341425669 (553h45m50.212508704s) [seg_pos={0 0}, upload=485513]: Segment reached optimal threshold of 13 pieces. Dec 28 17:14:50.341623144 (553h45m50.21270619s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection Dec 28 17:14:50.341630289 (553h45m50.212713324s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.341718456 (553h45m50.212801491s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 17:14:50.341736882 (553h45m50.212819927s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.342075917 (553h45m50.213158952s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: rpc: dial tcp 45.91.116.9:4264: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 17:14:50.342083492 (553h45m50.213166527s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.342168664 (553h45m50.213251709s) [seg_pos={0 0}, upload=485513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:14:50.342285436 (553h45m50.213368481s) [upload=485513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:14:50.342292880 (553h45m50.213375926s) [upload=485513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:14:50.628085582 (553h45m50.499168627s) [upload=485513]: Done uploading segments... Dec 28 17:14:50.628106132 (553h45m50.499189177s) [upload=485513]: Upload finished. err: <nil> Duration: 5.707296036s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X................] duration: 3.745878315s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:17:24 UTC) orbiter warning: context deadline exceeded Duration: 3.745910626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX..] duration: 22.963923751s trace: https://jaeger.dev.storj.io/trace/78b2fbe3857c51da FAIL (2025-12-28 17:17:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ) context deadline exceeded ======LOGS====== ec 28 17:14:47.122672129 (553h45m46.993755164s) [upload=485513]: Done starting upload Dec 28 17:14:47.122848355 (553h45m46.99393139s) [upload=485513]: Uploading segments... Dec 28 17:14:47.122940490 (553h45m46.994023525s) [upload=485513]: Got next segment. Inline: false Dec 28 17:14:47.122993632 (553h45m46.994076667s) [upload=485513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:14:47.123000024 (553h45m46.994083069s) [upload=485513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:14:47.366750740 (553h45m47.237833775s) [seg_pos={0 0}, upload=485513]: Begin upload segment... Dec 28 17:14:47.366798211 (553h45m47.237881256s) [seg_pos={0 0}, upload=485513]: Done begin upload segment. Dec 28 17:14:47.366833639 (553h45m47.237916674s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.366880908 (553h45m47.237963954s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.367791242 (553h45m47.238874287s) [noise=true , address=45.91.116.8:4046 , piece_id=NDZQO5FNABBKZVPB7WLPJKKXEFJCOW6WXFCSMQSU6G5O7ZOKJHWQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.367882796 (553h45m47.238965832s) [noise=true , address=45.91.116.13:4155 , piece_id=Y4R4FJGFL5UHC5GXSUL3HICW7D73VHCNYOYUI4NZZWVEKJYU3XDQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368120119 (553h45m47.239203154s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6KPTDQBX5XQWTNRJNN4LABJNMVSKSQGMXTEUKDXPL6BUWZLH5ZQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368655628 (553h45m47.239738663s) [noise=true , address=45.91.116.12:4225 , piece_id=ZX77FYRC46C2A7JO4BTKKMYL2RSDL3ME25SH4NS5TLDIK2Q6CINQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368879113 (553h45m47.239962158s) [noise=true , address=77.90.31.13:4629 , piece_id=2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.369518130 (553h45m47.240601165s) [noise=true , address=145.79.227.9:4366 , piece_id=MGYFRZHCXRP2LEN4DRFJJHQA6XMF5GCXUTEDFFDQOQ5LI4JC47CA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.369691581 (553h45m47.240774626s) [noise=true , address=145.79.227.8:4330 , piece_id=DRKFE3CPPP3CCMEQY63EQKYQXYU5PZPEJCIPCN3TO74LS7LZRT5A, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.370280673 (553h45m47.241363708s) [noise=true , address=145.79.227.15:4874 , piece_id=RDI2QQV23SKNFQT345S45FG35VQIUQWT3LYRAIOYLTNVDGHLMLLA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.370620850 (553h45m47.241703895s) [noise=true , address=145.79.227.14:4837 , piece_id=J2VEW6D6G7QQCBDSBNYUDWBQUTQY65J2SCO4UPNHDTX2B7XKSM3A, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371102617 (553h45m47.242185662s) [noise=true , address=145.79.227.32:4835 , piece_id=OYYIQY23ZCU2W3YU3MMF7F4YQJLD674YT5SMCHCS5RI5NCHR55FQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371447925 (553h45m47.24253097s) [noise=true , address=45.91.116.14:4191 , piece_id=HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371857875 (553h45m47.242940911s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.372547799 (553h45m47.243630834s) [noise=true , address=145.79.227.2:4115 , piece_id=OYPCRTHCLZPXDH4YRVZP3OMK7U4WIYUBCA4FCQ5TODI5COUZEJ7Q, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.373321012 (553h45m47.244404057s) [noise=true , address=145.79.227.16:4911 , piece_id=MCF3JNDKZR7C6OHM3YWN7RQQTKEYO657BJHPCMK34U44YK3E5ZVQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:49.084979743 (553h45m48.956062788s) [upload=485513]: Next returned nil segment Dec 28 17:14:49.084993810 (553h45m48.956076855s) [upload=485513]: Waiting for error group managing segments... Dec 28 17:14:49.370078022 (553h45m49.241161067s) [noise=true , address=145.79.227.15:4874 , piece_id=RDI2QQV23SKNFQT345S45FG35VQIUQWT3LYRAIOYLTNVDGHLMLLA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.370687373 (553h45m49.241770418s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6KPTDQBX5XQWTNRJNN4LABJNMVSKSQGMXTEUKDXPL6BUWZLH5ZQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.385089026 (553h45m49.256172072s) [noise=true , address=145.79.227.2:4115 , piece_id=OYPCRTHCLZPXDH4YRVZP3OMK7U4WIYUBCA4FCQ5TODI5COUZEJ7Q, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.387283836 (553h45m49.258366871s) [noise=true , address=145.79.227.9:4366 , piece_id=MGYFRZHCXRP2LEN4DRFJJHQA6XMF5GCXUTEDFFDQOQ5LI4JC47CA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.390730008 (553h45m49.261813053s) [noise=true , address=145.79.227.8:4330 , piece_id=DRKFE3CPPP3CCMEQY63EQKYQXYU5PZPEJCIPCN3TO74LS7LZRT5A, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.395841702 (553h45m49.266924737s) [noise=true , address=145.79.227.14:4837 , piece_id=J2VEW6D6G7QQCBDSBNYUDWBQUTQY65J2SCO4UPNHDTX2B7XKSM3A, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.402703177 (553h45m49.273786212s) [noise=true , address=145.79.227.16:4911 , piece_id=MCF3JNDKZR7C6OHM3YWN7RQQTKEYO657BJHPCMK34U44YK3E5ZVQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.408552185 (553h45m49.27963523s) [noise=true , address=145.79.227.32:4835 , piece_id=OYYIQY23ZCU2W3YU3MMF7F4YQJLD674YT5SMCHCS5RI5NCHR55FQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.588393496 (553h45m49.459476541s) [noise=true , address=45.91.116.14:4191 , piece_id=HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.615936172 (553h45m49.487019217s) [noise=true , address=77.90.31.13:4629 , piece_id=2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.751406332 (553h45m49.622489367s) [noise=true , address=45.91.116.8:4046 , piece_id=NDZQO5FNABBKZVPB7WLPJKKXEFJCOW6WXFCSMQSU6G5O7ZOKJHWQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.021029976 (553h45m49.892113011s) [noise=true , address=45.91.116.13:4155 , piece_id=Y4R4FJGFL5UHC5GXSUL3HICW7D73VHCNYOYUI4NZZWVEKJYU3XDQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.341406011 (553h45m50.212489056s) [noise=true , address=45.91.116.12:4225 , piece_id=ZX77FYRC46C2A7JO4BTKKMYL2RSDL3ME25SH4NS5TLDIK2Q6CINQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.341425669 (553h45m50.212508704s) [seg_pos={0 0}, upload=485513]: Segment reached optimal threshold of 13 pieces. Dec 28 17:14:50.341623144 (553h45m50.21270619s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection Dec 28 17:14:50.341630289 (553h45m50.212713324s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.341718456 (553h45m50.212801491s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 17:14:50.341736882 (553h45m50.212819927s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.342075917 (553h45m50.213158952s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: rpc: dial tcp 45.91.116.9:4264: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 17:14:50.342083492 (553h45m50.213166527s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.342168664 (553h45m50.213251709s) [seg_pos={0 0}, upload=485513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:14:50.342285436 (553h45m50.213368481s) [upload=485513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:14:50.342292880 (553h45m50.213375926s) [upload=485513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:14:50.628085582 (553h45m50.499168627s) [upload=485513]: Done uploading segments... Dec 28 17:14:50.628106132 (553h45m50.499189177s) [upload=485513]: Upload finished. err: <nil> Duration: 22.963956142s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X...............] duration: 3.377369273s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:17:01 UTC) orbiter warning: context deadline exceeded Duration: 3.377388201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X........................X.....................] duration: 35.477194708s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:16:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 35.477225035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X..............] duration: 3.444960898s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:16:32 UTC) orbiter warning: context deadline exceeded Duration: 3.444991186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX.] duration: 23.607504874s trace: https://jaeger.dev.storj.io/trace/78b2fbe3857c51da FAIL (2025-12-28 17:16:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ) context deadline exceeded ======LOGS====== ec 28 17:14:47.122672129 (553h45m46.993755164s) [upload=485513]: Done starting upload Dec 28 17:14:47.122848355 (553h45m46.99393139s) [upload=485513]: Uploading segments... Dec 28 17:14:47.122940490 (553h45m46.994023525s) [upload=485513]: Got next segment. Inline: false Dec 28 17:14:47.122993632 (553h45m46.994076667s) [upload=485513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:14:47.123000024 (553h45m46.994083069s) [upload=485513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:14:47.366750740 (553h45m47.237833775s) [seg_pos={0 0}, upload=485513]: Begin upload segment... Dec 28 17:14:47.366798211 (553h45m47.237881256s) [seg_pos={0 0}, upload=485513]: Done begin upload segment. Dec 28 17:14:47.366833639 (553h45m47.237916674s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.366880908 (553h45m47.237963954s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.367791242 (553h45m47.238874287s) [noise=true , address=45.91.116.8:4046 , piece_id=NDZQO5FNABBKZVPB7WLPJKKXEFJCOW6WXFCSMQSU6G5O7ZOKJHWQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.367882796 (553h45m47.238965832s) [noise=true , address=45.91.116.13:4155 , piece_id=Y4R4FJGFL5UHC5GXSUL3HICW7D73VHCNYOYUI4NZZWVEKJYU3XDQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368120119 (553h45m47.239203154s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6KPTDQBX5XQWTNRJNN4LABJNMVSKSQGMXTEUKDXPL6BUWZLH5ZQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368655628 (553h45m47.239738663s) [noise=true , address=45.91.116.12:4225 , piece_id=ZX77FYRC46C2A7JO4BTKKMYL2RSDL3ME25SH4NS5TLDIK2Q6CINQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368879113 (553h45m47.239962158s) [noise=true , address=77.90.31.13:4629 , piece_id=2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.369518130 (553h45m47.240601165s) [noise=true , address=145.79.227.9:4366 , piece_id=MGYFRZHCXRP2LEN4DRFJJHQA6XMF5GCXUTEDFFDQOQ5LI4JC47CA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.369691581 (553h45m47.240774626s) [noise=true , address=145.79.227.8:4330 , piece_id=DRKFE3CPPP3CCMEQY63EQKYQXYU5PZPEJCIPCN3TO74LS7LZRT5A, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.370280673 (553h45m47.241363708s) [noise=true , address=145.79.227.15:4874 , piece_id=RDI2QQV23SKNFQT345S45FG35VQIUQWT3LYRAIOYLTNVDGHLMLLA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.370620850 (553h45m47.241703895s) [noise=true , address=145.79.227.14:4837 , piece_id=J2VEW6D6G7QQCBDSBNYUDWBQUTQY65J2SCO4UPNHDTX2B7XKSM3A, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371102617 (553h45m47.242185662s) [noise=true , address=145.79.227.32:4835 , piece_id=OYYIQY23ZCU2W3YU3MMF7F4YQJLD674YT5SMCHCS5RI5NCHR55FQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371447925 (553h45m47.24253097s) [noise=true , address=45.91.116.14:4191 , piece_id=HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371857875 (553h45m47.242940911s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.372547799 (553h45m47.243630834s) [noise=true , address=145.79.227.2:4115 , piece_id=OYPCRTHCLZPXDH4YRVZP3OMK7U4WIYUBCA4FCQ5TODI5COUZEJ7Q, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.373321012 (553h45m47.244404057s) [noise=true , address=145.79.227.16:4911 , piece_id=MCF3JNDKZR7C6OHM3YWN7RQQTKEYO657BJHPCMK34U44YK3E5ZVQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:49.084979743 (553h45m48.956062788s) [upload=485513]: Next returned nil segment Dec 28 17:14:49.084993810 (553h45m48.956076855s) [upload=485513]: Waiting for error group managing segments... Dec 28 17:14:49.370078022 (553h45m49.241161067s) [noise=true , address=145.79.227.15:4874 , piece_id=RDI2QQV23SKNFQT345S45FG35VQIUQWT3LYRAIOYLTNVDGHLMLLA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.370687373 (553h45m49.241770418s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6KPTDQBX5XQWTNRJNN4LABJNMVSKSQGMXTEUKDXPL6BUWZLH5ZQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.385089026 (553h45m49.256172072s) [noise=true , address=145.79.227.2:4115 , piece_id=OYPCRTHCLZPXDH4YRVZP3OMK7U4WIYUBCA4FCQ5TODI5COUZEJ7Q, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.387283836 (553h45m49.258366871s) [noise=true , address=145.79.227.9:4366 , piece_id=MGYFRZHCXRP2LEN4DRFJJHQA6XMF5GCXUTEDFFDQOQ5LI4JC47CA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.390730008 (553h45m49.261813053s) [noise=true , address=145.79.227.8:4330 , piece_id=DRKFE3CPPP3CCMEQY63EQKYQXYU5PZPEJCIPCN3TO74LS7LZRT5A, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.395841702 (553h45m49.266924737s) [noise=true , address=145.79.227.14:4837 , piece_id=J2VEW6D6G7QQCBDSBNYUDWBQUTQY65J2SCO4UPNHDTX2B7XKSM3A, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.402703177 (553h45m49.273786212s) [noise=true , address=145.79.227.16:4911 , piece_id=MCF3JNDKZR7C6OHM3YWN7RQQTKEYO657BJHPCMK34U44YK3E5ZVQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.408552185 (553h45m49.27963523s) [noise=true , address=145.79.227.32:4835 , piece_id=OYYIQY23ZCU2W3YU3MMF7F4YQJLD674YT5SMCHCS5RI5NCHR55FQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.588393496 (553h45m49.459476541s) [noise=true , address=45.91.116.14:4191 , piece_id=HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.615936172 (553h45m49.487019217s) [noise=true , address=77.90.31.13:4629 , piece_id=2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.751406332 (553h45m49.622489367s) [noise=true , address=45.91.116.8:4046 , piece_id=NDZQO5FNABBKZVPB7WLPJKKXEFJCOW6WXFCSMQSU6G5O7ZOKJHWQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.021029976 (553h45m49.892113011s) [noise=true , address=45.91.116.13:4155 , piece_id=Y4R4FJGFL5UHC5GXSUL3HICW7D73VHCNYOYUI4NZZWVEKJYU3XDQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.341406011 (553h45m50.212489056s) [noise=true , address=45.91.116.12:4225 , piece_id=ZX77FYRC46C2A7JO4BTKKMYL2RSDL3ME25SH4NS5TLDIK2Q6CINQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.341425669 (553h45m50.212508704s) [seg_pos={0 0}, upload=485513]: Segment reached optimal threshold of 13 pieces. Dec 28 17:14:50.341623144 (553h45m50.21270619s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection Dec 28 17:14:50.341630289 (553h45m50.212713324s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.341718456 (553h45m50.212801491s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 17:14:50.341736882 (553h45m50.212819927s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.342075917 (553h45m50.213158952s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: rpc: dial tcp 45.91.116.9:4264: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 17:14:50.342083492 (553h45m50.213166527s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.342168664 (553h45m50.213251709s) [seg_pos={0 0}, upload=485513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:14:50.342285436 (553h45m50.213368481s) [upload=485513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:14:50.342292880 (553h45m50.213375926s) [upload=485513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:14:50.628085582 (553h45m50.499168627s) [upload=485513]: Done uploading segments... Dec 28 17:14:50.628106132 (553h45m50.499189177s) [upload=485513]: Upload finished. err: <nil> Duration: 23.607537117s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X.............] duration: 4.152711177s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:15:52 UTC) orbiter warning: context deadline exceeded Duration: 4.152755571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X........................X....................] duration: 16.005535393s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 16.005562724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX] duration: 30.003081688s trace: https://jaeger.dev.storj.io/trace/78b2fbe3857c51da FAIL (2025-12-28 17:15:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ) context deadline exceeded ======LOGS====== ec 28 17:14:47.122672129 (553h45m46.993755164s) [upload=485513]: Done starting upload Dec 28 17:14:47.122848355 (553h45m46.99393139s) [upload=485513]: Uploading segments... Dec 28 17:14:47.122940490 (553h45m46.994023525s) [upload=485513]: Got next segment. Inline: false Dec 28 17:14:47.122993632 (553h45m46.994076667s) [upload=485513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:14:47.123000024 (553h45m46.994083069s) [upload=485513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:14:47.366750740 (553h45m47.237833775s) [seg_pos={0 0}, upload=485513]: Begin upload segment... Dec 28 17:14:47.366798211 (553h45m47.237881256s) [seg_pos={0 0}, upload=485513]: Done begin upload segment. Dec 28 17:14:47.366833639 (553h45m47.237916674s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.366880908 (553h45m47.237963954s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.367791242 (553h45m47.238874287s) [noise=true , address=45.91.116.8:4046 , piece_id=NDZQO5FNABBKZVPB7WLPJKKXEFJCOW6WXFCSMQSU6G5O7ZOKJHWQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.367882796 (553h45m47.238965832s) [noise=true , address=45.91.116.13:4155 , piece_id=Y4R4FJGFL5UHC5GXSUL3HICW7D73VHCNYOYUI4NZZWVEKJYU3XDQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368120119 (553h45m47.239203154s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6KPTDQBX5XQWTNRJNN4LABJNMVSKSQGMXTEUKDXPL6BUWZLH5ZQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368655628 (553h45m47.239738663s) [noise=true , address=45.91.116.12:4225 , piece_id=ZX77FYRC46C2A7JO4BTKKMYL2RSDL3ME25SH4NS5TLDIK2Q6CINQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.368879113 (553h45m47.239962158s) [noise=true , address=77.90.31.13:4629 , piece_id=2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.369518130 (553h45m47.240601165s) [noise=true , address=145.79.227.9:4366 , piece_id=MGYFRZHCXRP2LEN4DRFJJHQA6XMF5GCXUTEDFFDQOQ5LI4JC47CA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.369691581 (553h45m47.240774626s) [noise=true , address=145.79.227.8:4330 , piece_id=DRKFE3CPPP3CCMEQY63EQKYQXYU5PZPEJCIPCN3TO74LS7LZRT5A, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.370280673 (553h45m47.241363708s) [noise=true , address=145.79.227.15:4874 , piece_id=RDI2QQV23SKNFQT345S45FG35VQIUQWT3LYRAIOYLTNVDGHLMLLA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.370620850 (553h45m47.241703895s) [noise=true , address=145.79.227.14:4837 , piece_id=J2VEW6D6G7QQCBDSBNYUDWBQUTQY65J2SCO4UPNHDTX2B7XKSM3A, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371102617 (553h45m47.242185662s) [noise=true , address=145.79.227.32:4835 , piece_id=OYYIQY23ZCU2W3YU3MMF7F4YQJLD674YT5SMCHCS5RI5NCHR55FQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371447925 (553h45m47.24253097s) [noise=true , address=45.91.116.14:4191 , piece_id=HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.371857875 (553h45m47.242940911s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.372547799 (553h45m47.243630834s) [noise=true , address=145.79.227.2:4115 , piece_id=OYPCRTHCLZPXDH4YRVZP3OMK7U4WIYUBCA4FCQ5TODI5COUZEJ7Q, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:47.373321012 (553h45m47.244404057s) [noise=true , address=145.79.227.16:4911 , piece_id=MCF3JNDKZR7C6OHM3YWN7RQQTKEYO657BJHPCMK34U44YK3E5ZVQ, seg_pos={0 0}, upload=485513]: Uploading piece... Dec 28 17:14:49.084979743 (553h45m48.956062788s) [upload=485513]: Next returned nil segment Dec 28 17:14:49.084993810 (553h45m48.956076855s) [upload=485513]: Waiting for error group managing segments... Dec 28 17:14:49.370078022 (553h45m49.241161067s) [noise=true , address=145.79.227.15:4874 , piece_id=RDI2QQV23SKNFQT345S45FG35VQIUQWT3LYRAIOYLTNVDGHLMLLA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.370687373 (553h45m49.241770418s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6KPTDQBX5XQWTNRJNN4LABJNMVSKSQGMXTEUKDXPL6BUWZLH5ZQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.385089026 (553h45m49.256172072s) [noise=true , address=145.79.227.2:4115 , piece_id=OYPCRTHCLZPXDH4YRVZP3OMK7U4WIYUBCA4FCQ5TODI5COUZEJ7Q, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.387283836 (553h45m49.258366871s) [noise=true , address=145.79.227.9:4366 , piece_id=MGYFRZHCXRP2LEN4DRFJJHQA6XMF5GCXUTEDFFDQOQ5LI4JC47CA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.390730008 (553h45m49.261813053s) [noise=true , address=145.79.227.8:4330 , piece_id=DRKFE3CPPP3CCMEQY63EQKYQXYU5PZPEJCIPCN3TO74LS7LZRT5A, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.395841702 (553h45m49.266924737s) [noise=true , address=145.79.227.14:4837 , piece_id=J2VEW6D6G7QQCBDSBNYUDWBQUTQY65J2SCO4UPNHDTX2B7XKSM3A, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.402703177 (553h45m49.273786212s) [noise=true , address=145.79.227.16:4911 , piece_id=MCF3JNDKZR7C6OHM3YWN7RQQTKEYO657BJHPCMK34U44YK3E5ZVQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.408552185 (553h45m49.27963523s) [noise=true , address=145.79.227.32:4835 , piece_id=OYYIQY23ZCU2W3YU3MMF7F4YQJLD674YT5SMCHCS5RI5NCHR55FQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.588393496 (553h45m49.459476541s) [noise=true , address=45.91.116.14:4191 , piece_id=HIONGRMD2535GKHBU3NIDNW3I5OXZTCXYPB2N2DPZJSADST24A6Q, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.615936172 (553h45m49.487019217s) [noise=true , address=77.90.31.13:4629 , piece_id=2H5HFCLYRNKWTKEH3M7M2LIXCMF6K6KP4LVA6QGSIESVB6BGI6CQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:49.751406332 (553h45m49.622489367s) [noise=true , address=45.91.116.8:4046 , piece_id=NDZQO5FNABBKZVPB7WLPJKKXEFJCOW6WXFCSMQSU6G5O7ZOKJHWQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.021029976 (553h45m49.892113011s) [noise=true , address=45.91.116.13:4155 , piece_id=Y4R4FJGFL5UHC5GXSUL3HICW7D73VHCNYOYUI4NZZWVEKJYU3XDQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.341406011 (553h45m50.212489056s) [noise=true , address=45.91.116.12:4225 , piece_id=ZX77FYRC46C2A7JO4BTKKMYL2RSDL3ME25SH4NS5TLDIK2Q6CINQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: <nil> Dec 28 17:14:50.341425669 (553h45m50.212508704s) [seg_pos={0 0}, upload=485513]: Segment reached optimal threshold of 13 pieces. Dec 28 17:14:50.341623144 (553h45m50.21270619s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47980->145.79.227.4:4186: use of closed network connection Dec 28 17:14:50.341630289 (553h45m50.212713324s) [noise=true , address=145.79.227.4:4186 , piece_id=ZZYRQI4QGIZLTKT2EVDXXR5OOD37565TIFJN76W6R2FAVAZCOEGQ, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.341718456 (553h45m50.212801491s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 17:14:50.341736882 (553h45m50.212819927s) [noise=true , address=145.79.227.29:4941 , piece_id=LDXD2DOE4GKSR2SIUFHJCGIDTCX6TI4OD7UIGEEQYG27EVRVLZCA, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.342075917 (553h45m50.213158952s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: rpc: dial tcp 45.91.116.9:4264: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 17:14:50.342083492 (553h45m50.213166527s) [noise=true , address=45.91.116.9:4264 , piece_id=QMX4E4HFKRMLKUG7EHOJ6V5JEIGTTXAY54LUIEDPMFCEL2MRRWMQ, seg_pos={0 0}, upload=485513]: Long tail cancellation due to optimal threshold Dec 28 17:14:50.342168664 (553h45m50.213251709s) [seg_pos={0 0}, upload=485513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:14:50.342285436 (553h45m50.213368481s) [upload=485513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:14:50.342292880 (553h45m50.213375926s) [upload=485513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:14:50.628085582 (553h45m50.499168627s) [upload=485513]: Done uploading segments... Dec 28 17:14:50.628106132 (553h45m50.499189177s) [upload=485513]: Upload finished. err: <nil> Duration: 30.003103369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X........................X...................] duration: 27.922666339s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:14:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 27.922699573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X............] duration: 4.038817334s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:14:43 UTC) orbiter warning: context deadline exceeded Duration: 4.038855406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X........................X..................] duration: 47.468996375s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 47.4690247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...X] duration: 30.003414402s trace: https://jaeger.dev.storj.io/trace/90298eb3e9ade85 FAIL (2025-12-28 17:14:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 17:13:46.429342579 (553h44m46.300425614s) [upload=485501]: Starting upload Dec 28 17:13:46.429411230 (553h44m46.300494265s) [upload=485501]: Done starting upload Dec 28 17:13:46.429445505 (553h44m46.30052854s) [upload=485501]: Uploading segments... Dec 28 17:13:46.429636438 (553h44m46.300719474s) [upload=485501]: Got next segment. Inline: false Dec 28 17:13:46.429678929 (553h44m46.300761974s) [upload=485501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:13:46.429684339 (553h44m46.300767374s) [upload=485501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:13:46.680124834 (553h44m46.551207879s) [seg_pos={0 0}, upload=485501]: Begin upload segment... Dec 28 17:13:46.680202732 (553h44m46.551285767s) [seg_pos={0 0}, upload=485501]: Done begin upload segment. Dec 28 17:13:46.680248850 (553h44m46.551331886s) [noise=true , address=145.79.227.52:4447 , piece_id=NUAFT3KCHND65S5HNIINP5YWDMS6AUL6ZFX2YDN6U75OMXIGHRVQ, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.680371203 (553h44m46.551454248s) [noise=true , address=45.91.116.11:4120 , piece_id=IJIVRPY7YH2RNMWYQHLVVK4OGBUGHUS6EZ4GCGMJHLO33UGK7FWQ, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.680893568 (553h44m46.551976603s) [noise=true , address=77.90.31.11:4556 , piece_id=P5YKGYNNQJ7NTQ5KGMJA7ZN5766PGLS4PCERR4G56KDJ5H46RWYA, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.681199641 (553h44m46.552282676s) [noise=true , address=145.79.227.39:4015 , piece_id=VEFFW755JOBPX3T2COHDSFW4P3PJQL4TNKOA3GMC3P4YCAMEPQ2A, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.681511114 (553h44m46.552594159s) [noise=true , address=145.79.227.11:4732 , piece_id=RSN5FXCNGPM4PNM7LGCRKVSULJUQNQB6HTLR5LUMZDBIB5TKPPHA, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.681785947 (553h44m46.552868982s) [noise=true , address=145.79.227.17:4971 , piece_id=NHPEOOWBYVCSFAO4XZ5BHC2KP5FOCHFXJC2TR5DPF2WQUHCR7FXQ, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.682262605 (553h44m46.55334564s) [noise=true , address=45.91.116.14:4189 , piece_id=CX7WEFY6YWY6FVEKY2OJOQ652NHDJKQC4NKDXBBVEO3QDU5XADVQ, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.683640158 (553h44m46.554723193s) [noise=true , address=145.79.227.46:4601 , piece_id=C7LOPPTGVYNAMKCXSRHEHYJEY53TH3FQ5IYMVCE6IU4ARDM4NNXA, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.684869189 (553h44m46.555952224s) [noise=true , address=145.79.227.36:4519 , piece_id=Z7DFEJDYM37DMN7DXRY3J75XNCOOQZYTMIRVFSQFQ47RLCOFS2HQ, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.687039190 (553h44m46.558122225s) [noise=true , address=45.91.116.12:4228 , piece_id=KJKAQS6VU56FZAYYBQQ2YY5U4OZVVJY7AISQNI2YIXBIMLTWUEKQ, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.688441431 (553h44m46.559524476s) [noise=true , address=145.79.227.37:4566 , piece_id=NBGFI3QLY5TXS4GU26NB7SKRRGFJAET762YA7DL6WMOAXPUJKJJA, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.690207924 (553h44m46.561290959s) [noise=true , address=45.91.116.7:4012 , piece_id=BY7G32EXDDXWO7A7QBYW36MZ23SEDILK7NJXA3RFNVCOAVHFX3KQ, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.690362840 (553h44m46.561445875s) [noise=true , address=45.91.116.5:4300 , piece_id=AKJUDU3LRRXPJWZASV57OK5SYQBDVUFOXB7RBHPJS5OPJCSKZGPA, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.691924473 (553h44m46.563007509s) [noise=true , address=145.79.227.42:4122 , piece_id=QGJCYR2S3Y33UQBCYYK5KUO54F7K6DY6LRDZ573XNUN5D5WFBRIQ, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.692495009 (553h44m46.563578045s) [noise=true , address=145.79.227.5:4223 , piece_id=VWBSQME2NHOL7V5GLLUQ5J3XDJOY77EBCVVM4I3MG5ZIWPFE524A, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:46.693870289 (553h44m46.564953354s) [noise=true , address=145.79.227.2:4115 , piece_id=C4GL36GQRVJ4FQSIGCZBIABHVKFQQAVNP27IX6QXMWYXC5JTXN7A, seg_pos={0 0}, upload=485501]: Uploading piece... Dec 28 17:13:48.356450598 (553h44m48.227533643s) [upload=485501]: Next returned nil segment Dec 28 17:13:48.356498539 (553h44m48.227581584s) [upload=485501]: Waiting for error group managing segments... Dec 28 17:13:48.685952630 (553h44m48.557035665s) [noise=true , address=145.79.227.39:4015 , piece_id=VEFFW755JOBPX3T2COHDSFW4P3PJQL4TNKOA3GMC3P4YCAMEPQ2A, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.687970433 (553h44m48.559053468s) [noise=true , address=145.79.227.42:4122 , piece_id=QGJCYR2S3Y33UQBCYYK5KUO54F7K6DY6LRDZ573XNUN5D5WFBRIQ, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.689456713 (553h44m48.560539748s) [noise=true , address=145.79.227.46:4601 , piece_id=C7LOPPTGVYNAMKCXSRHEHYJEY53TH3FQ5IYMVCE6IU4ARDM4NNXA, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.690032559 (553h44m48.561115594s) [noise=true , address=145.79.227.17:4971 , piece_id=NHPEOOWBYVCSFAO4XZ5BHC2KP5FOCHFXJC2TR5DPF2WQUHCR7FXQ, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.699036915 (553h44m48.57011995s) [noise=true , address=145.79.227.36:4519 , piece_id=Z7DFEJDYM37DMN7DXRY3J75XNCOOQZYTMIRVFSQFQ47RLCOFS2HQ, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.699855103 (553h44m48.570938138s) [noise=true , address=145.79.227.11:4732 , piece_id=RSN5FXCNGPM4PNM7LGCRKVSULJUQNQB6HTLR5LUMZDBIB5TKPPHA, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.706924305 (553h44m48.57800735s) [noise=true , address=145.79.227.5:4223 , piece_id=VWBSQME2NHOL7V5GLLUQ5J3XDJOY77EBCVVM4I3MG5ZIWPFE524A, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.708585228 (553h44m48.579668263s) [noise=true , address=145.79.227.52:4447 , piece_id=NUAFT3KCHND65S5HNIINP5YWDMS6AUL6ZFX2YDN6U75OMXIGHRVQ, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.708830494 (553h44m48.57991353s) [noise=true , address=145.79.227.37:4566 , piece_id=NBGFI3QLY5TXS4GU26NB7SKRRGFJAET762YA7DL6WMOAXPUJKJJA, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.923120662 (553h44m48.794203707s) [noise=true , address=45.91.116.11:4120 , piece_id=IJIVRPY7YH2RNMWYQHLVVK4OGBUGHUS6EZ4GCGMJHLO33UGK7FWQ, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.964434832 (553h44m48.835517867s) [noise=true , address=45.91.116.5:4300 , piece_id=AKJUDU3LRRXPJWZASV57OK5SYQBDVUFOXB7RBHPJS5OPJCSKZGPA, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:48.973931164 (553h44m48.8450142s) [noise=true , address=77.90.31.11:4556 , piece_id=P5YKGYNNQJ7NTQ5KGMJA7ZN5766PGLS4PCERR4G56KDJ5H46RWYA, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:49.043577675 (553h44m48.91466072s) [noise=true , address=45.91.116.14:4189 , piece_id=CX7WEFY6YWY6FVEKY2OJOQ652NHDJKQC4NKDXBBVEO3QDU5XADVQ, seg_pos={0 0}, upload=485501]: Done uploading piece. err: <nil> Dec 28 17:13:49.043603033 (553h44m48.914686078s) [seg_pos={0 0}, upload=485501]: Segment reached optimal threshold of 13 pieces. Dec 28 17:13:49.043841007 (553h44m48.914924052s) [noise=true , address=45.91.116.7:4012 , piece_id=BY7G32EXDDXWO7A7QBYW36MZ23SEDILK7NJXA3RFNVCOAVHFX3KQ, seg_pos={0 0}, upload=485501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 17:13:49.043851657 (553h44m48.914934702s) [noise=true , address=45.91.116.7:4012 , piece_id=BY7G32EXDDXWO7A7QBYW36MZ23SEDILK7NJXA3RFNVCOAVHFX3KQ, seg_pos={0 0}, upload=485501]: Long tail cancellation due to optimal threshold Dec 28 17:13:49.044043081 (553h44m48.915126116s) [noise=true , address=45.91.116.12:4228 , piece_id=KJKAQS6VU56FZAYYBQQ2YY5U4OZVVJY7AISQNI2YIXBIMLTWUEKQ, seg_pos={0 0}, upload=485501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 17:13:49.044080402 (553h44m48.915163437s) [noise=true , address=45.91.116.12:4228 , piece_id=KJKAQS6VU56FZAYYBQQ2YY5U4OZVVJY7AISQNI2YIXBIMLTWUEKQ, seg_pos={0 0}, upload=485501]: Long tail cancellation due to optimal threshold Dec 28 17:13:49.044114927 (553h44m48.915197972s) [noise=true , address=145.79.227.2:4115 , piece_id=C4GL36GQRVJ4FQSIGCZBIABHVKFQQAVNP27IX6QXMWYXC5JTXN7A, seg_pos={0 0}, upload=485501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:52276->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:52276->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->145.79.227.2:4115: use of closed network connection Dec 28 17:13:49.044124426 (553h44m48.915207471s) [noise=true , address=145.79.227.2:4115 , piece_id=C4GL36GQRVJ4FQSIGCZBIABHVKFQQAVNP27IX6QXMWYXC5JTXN7A, seg_pos={0 0}, upload=485501]: Long tail cancellation due to optimal threshold Dec 28 17:13:49.044244334 (553h44m48.915327369s) [seg_pos={0 0}, upload=485501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:13:49.044346268 (553h44m48.915429313s) [upload=485501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:13:49.044352099 (553h44m48.915435134s) [upload=485501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:13:49.342754593 (553h44m49.213837638s) [upload=485501]: Done uploading segments... Dec 28 17:13:49.342777166 (553h44m49.213860201s) [upload=485501]: Upload finished. err: <nil> Duration: 30.003435842s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X...........] duration: 3.442145424s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:13:32 UTC) orbiter warning: context deadline exceeded Duration: 3.44217418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X........................X.................] duration: 30.703508022s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:13:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 30.703526627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX...] duration: 6.016404315s trace: https://jaeger.dev.storj.io/trace/7ee1234b4dd3f170 FAIL (2025-12-28 17:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ) context deadline exceeded ======LOGS====== Dec 28 17:10:31.923915099 (553h41m31.794998134s) [upload=485448]: Starting upload Dec 28 17:10:31.923984641 (553h41m31.795067676s) [upload=485448]: Done starting upload Dec 28 17:10:31.924099270 (553h41m31.795182315s) [upload=485448]: Uploading segments... Dec 28 17:10:31.924177739 (553h41m31.795260784s) [upload=485448]: Got next segment. Inline: false Dec 28 17:10:31.924213748 (553h41m31.795296783s) [upload=485448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:10:31.924218767 (553h41m31.795301802s) [upload=485448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:10:32.171366966 (553h41m32.042450011s) [seg_pos={0 0}, upload=485448]: Begin upload segment... Dec 28 17:10:32.171441368 (553h41m32.042524413s) [seg_pos={0 0}, upload=485448]: Done begin upload segment. Dec 28 17:10:32.171484040 (553h41m32.042567085s) [noise=true , address=145.79.227.50:4374 , piece_id=Q72FSMVVPPCFCCIBIHAUUZUIQ3TB42XGA2AFBHYFESO3Z6HG2DEA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.173952240 (553h41m32.045035275s) [noise=true , address=145.79.227.46:4603 , piece_id=5OLCL5V2EBUBTOOAERF2UILITWPEPYK7HNYT6U7B7PRYAJ4PUYVA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.174429098 (553h41m32.045512143s) [noise=true , address=45.91.116.4:4937 , piece_id=DRIAIJZNCVK72UWNHN26J7OXJGGWBBGYFDB24SS36VI4XFOFQFPA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.174724190 (553h41m32.045807225s) [noise=true , address=145.79.227.8:4330 , piece_id=G6BYDKHAIKK7EPVXX2P7KNOFXTCICMPN53DAITHH4L6LRAV65EGA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175039961 (553h41m32.046122996s) [noise=true , address=145.79.227.37:4566 , piece_id=UZCWMZHMXENHLHZXYVKRDWEHPIORV4FZRXYY3HDE2IAASQUXZVMA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175358577 (553h41m32.046441613s) [noise=true , address=145.79.227.39:4013 , piece_id=F55YK4E643BOGT7IVHIVBICZ5HLZCXIM2HHOUZUBYZ3V6ZIBETOA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175685741 (553h41m32.046768776s) [noise=true , address=145.79.227.38:4977 , piece_id=MTWFD4H5YVKN3G7NFQZ7TDLSSOFKWXCYRBB6ALYKSSRUCBEIHD2A, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.176729238 (553h41m32.047812273s) [noise=true , address=45.91.116.8:4048 , piece_id=P37SE2DCIJSOETG66LEY2RXCIEE52UKOA2N5SPF27IOYITAOCBNQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.176863583 (553h41m32.047946628s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.178256195 (553h41m32.04933923s) [noise=true , address=77.90.31.11:4554 , piece_id=DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.179005732 (553h41m32.050088768s) [noise=true , address=145.79.227.51:4411 , piece_id=QO3IXN3WINZNGI5XPKXJNG7EAGVBNHAPU2ADVCFMW3UN76MY4RBQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.182431747 (553h41m32.053514782s) [noise=true , address=45.91.116.13:4155 , piece_id=IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.185593568 (553h41m32.056676603s) [noise=true , address=145.79.227.52:4446 , piece_id=AA2GMRF6XXJFHDQ3KMYEWE7CCQYBP4LWDWNPA2SMSMUXPHRMIHIQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.186997140 (553h41m32.058080175s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.189407279 (553h41m32.060490354s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.194868729 (553h41m32.065951774s) [noise=true , address=145.79.227.47:4266 , piece_id=6HQH62HPXMIKR4AXMNUWF74JBBN3756KYEFVLX5BSVJ5THE5TS5Q, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:33.896978888 (553h41m33.768061963s) [upload=485448]: Next returned nil segment Dec 28 17:10:33.897003083 (553h41m33.768086129s) [upload=485448]: Waiting for error group managing segments... Dec 28 17:10:34.185240215 (553h41m34.05632326s) [noise=true , address=145.79.227.38:4977 , piece_id=MTWFD4H5YVKN3G7NFQZ7TDLSSOFKWXCYRBB6ALYKSSRUCBEIHD2A, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.190704710 (553h41m34.061787755s) [noise=true , address=145.79.227.51:4411 , piece_id=QO3IXN3WINZNGI5XPKXJNG7EAGVBNHAPU2ADVCFMW3UN76MY4RBQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.193934221 (553h41m34.065017256s) [noise=true , address=145.79.227.39:4013 , piece_id=F55YK4E643BOGT7IVHIVBICZ5HLZCXIM2HHOUZUBYZ3V6ZIBETOA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.194813936 (553h41m34.065896981s) [noise=true , address=145.79.227.50:4374 , piece_id=Q72FSMVVPPCFCCIBIHAUUZUIQ3TB42XGA2AFBHYFESO3Z6HG2DEA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.208863379 (553h41m34.079946414s) [noise=true , address=145.79.227.46:4603 , piece_id=5OLCL5V2EBUBTOOAERF2UILITWPEPYK7HNYT6U7B7PRYAJ4PUYVA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.209441920 (553h41m34.080524965s) [noise=true , address=145.79.227.47:4266 , piece_id=6HQH62HPXMIKR4AXMNUWF74JBBN3756KYEFVLX5BSVJ5THE5TS5Q, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.213251424 (553h41m34.084334459s) [noise=true , address=145.79.227.8:4330 , piece_id=G6BYDKHAIKK7EPVXX2P7KNOFXTCICMPN53DAITHH4L6LRAV65EGA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.217652074 (553h41m34.088735109s) [noise=true , address=145.79.227.37:4566 , piece_id=UZCWMZHMXENHLHZXYVKRDWEHPIORV4FZRXYY3HDE2IAASQUXZVMA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.238730301 (553h41m34.109813336s) [noise=true , address=145.79.227.52:4446 , piece_id=AA2GMRF6XXJFHDQ3KMYEWE7CCQYBP4LWDWNPA2SMSMUXPHRMIHIQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.425277103 (553h41m34.296360148s) [noise=true , address=77.90.31.11:4554 , piece_id=DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.515389765 (553h41m34.3864728s) [noise=true , address=45.91.116.4:4937 , piece_id=DRIAIJZNCVK72UWNHN26J7OXJGGWBBGYFDB24SS36VI4XFOFQFPA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.516022371 (553h41m34.387105416s) [noise=true , address=45.91.116.8:4048 , piece_id=P37SE2DCIJSOETG66LEY2RXCIEE52UKOA2N5SPF27IOYITAOCBNQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:35.121095662 (553h41m34.992178707s) [noise=true , address=45.91.116.13:4155 , piece_id=IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:35.121121400 (553h41m34.992204435s) [seg_pos={0 0}, upload=485448]: Segment reached optimal threshold of 13 pieces. Dec 28 17:10:35.121316322 (553h41m34.992399367s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.121323455 (553h41m34.99240649s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.121641731 (553h41m34.992724766s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.121648545 (553h41m34.99273159s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.122210485 (553h41m34.99329353s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.122227136 (553h41m34.993310181s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.122252895 (553h41m34.99333593s) [seg_pos={0 0}, upload=485448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:10:35.122382432 (553h41m34.993465477s) [upload=485448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:10:35.122388914 (553h41m34.993471959s) [upload=485448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:10:35.452563433 (553h41m35.323646478s) [upload=485448]: Done uploading segments... Dec 28 17:10:35.452604561 (553h41m35.323687606s) [upload=485448]: Upload finished. err: <nil> Duration: 6.016432147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X........................X................] duration: 48.280451314s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:12:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 48.280467423s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X..........] duration: 3.528105724s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:12:32 UTC) orbiter warning: context deadline exceeded Duration: 3.528180346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX..] duration: 6.649252678s trace: https://jaeger.dev.storj.io/trace/7ee1234b4dd3f170 FAIL (2025-12-28 17:12:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ) context deadline exceeded ======LOGS====== Dec 28 17:10:31.923915099 (553h41m31.794998134s) [upload=485448]: Starting upload Dec 28 17:10:31.923984641 (553h41m31.795067676s) [upload=485448]: Done starting upload Dec 28 17:10:31.924099270 (553h41m31.795182315s) [upload=485448]: Uploading segments... Dec 28 17:10:31.924177739 (553h41m31.795260784s) [upload=485448]: Got next segment. Inline: false Dec 28 17:10:31.924213748 (553h41m31.795296783s) [upload=485448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:10:31.924218767 (553h41m31.795301802s) [upload=485448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:10:32.171366966 (553h41m32.042450011s) [seg_pos={0 0}, upload=485448]: Begin upload segment... Dec 28 17:10:32.171441368 (553h41m32.042524413s) [seg_pos={0 0}, upload=485448]: Done begin upload segment. Dec 28 17:10:32.171484040 (553h41m32.042567085s) [noise=true , address=145.79.227.50:4374 , piece_id=Q72FSMVVPPCFCCIBIHAUUZUIQ3TB42XGA2AFBHYFESO3Z6HG2DEA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.173952240 (553h41m32.045035275s) [noise=true , address=145.79.227.46:4603 , piece_id=5OLCL5V2EBUBTOOAERF2UILITWPEPYK7HNYT6U7B7PRYAJ4PUYVA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.174429098 (553h41m32.045512143s) [noise=true , address=45.91.116.4:4937 , piece_id=DRIAIJZNCVK72UWNHN26J7OXJGGWBBGYFDB24SS36VI4XFOFQFPA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.174724190 (553h41m32.045807225s) [noise=true , address=145.79.227.8:4330 , piece_id=G6BYDKHAIKK7EPVXX2P7KNOFXTCICMPN53DAITHH4L6LRAV65EGA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175039961 (553h41m32.046122996s) [noise=true , address=145.79.227.37:4566 , piece_id=UZCWMZHMXENHLHZXYVKRDWEHPIORV4FZRXYY3HDE2IAASQUXZVMA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175358577 (553h41m32.046441613s) [noise=true , address=145.79.227.39:4013 , piece_id=F55YK4E643BOGT7IVHIVBICZ5HLZCXIM2HHOUZUBYZ3V6ZIBETOA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175685741 (553h41m32.046768776s) [noise=true , address=145.79.227.38:4977 , piece_id=MTWFD4H5YVKN3G7NFQZ7TDLSSOFKWXCYRBB6ALYKSSRUCBEIHD2A, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.176729238 (553h41m32.047812273s) [noise=true , address=45.91.116.8:4048 , piece_id=P37SE2DCIJSOETG66LEY2RXCIEE52UKOA2N5SPF27IOYITAOCBNQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.176863583 (553h41m32.047946628s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.178256195 (553h41m32.04933923s) [noise=true , address=77.90.31.11:4554 , piece_id=DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.179005732 (553h41m32.050088768s) [noise=true , address=145.79.227.51:4411 , piece_id=QO3IXN3WINZNGI5XPKXJNG7EAGVBNHAPU2ADVCFMW3UN76MY4RBQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.182431747 (553h41m32.053514782s) [noise=true , address=45.91.116.13:4155 , piece_id=IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.185593568 (553h41m32.056676603s) [noise=true , address=145.79.227.52:4446 , piece_id=AA2GMRF6XXJFHDQ3KMYEWE7CCQYBP4LWDWNPA2SMSMUXPHRMIHIQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.186997140 (553h41m32.058080175s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.189407279 (553h41m32.060490354s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.194868729 (553h41m32.065951774s) [noise=true , address=145.79.227.47:4266 , piece_id=6HQH62HPXMIKR4AXMNUWF74JBBN3756KYEFVLX5BSVJ5THE5TS5Q, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:33.896978888 (553h41m33.768061963s) [upload=485448]: Next returned nil segment Dec 28 17:10:33.897003083 (553h41m33.768086129s) [upload=485448]: Waiting for error group managing segments... Dec 28 17:10:34.185240215 (553h41m34.05632326s) [noise=true , address=145.79.227.38:4977 , piece_id=MTWFD4H5YVKN3G7NFQZ7TDLSSOFKWXCYRBB6ALYKSSRUCBEIHD2A, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.190704710 (553h41m34.061787755s) [noise=true , address=145.79.227.51:4411 , piece_id=QO3IXN3WINZNGI5XPKXJNG7EAGVBNHAPU2ADVCFMW3UN76MY4RBQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.193934221 (553h41m34.065017256s) [noise=true , address=145.79.227.39:4013 , piece_id=F55YK4E643BOGT7IVHIVBICZ5HLZCXIM2HHOUZUBYZ3V6ZIBETOA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.194813936 (553h41m34.065896981s) [noise=true , address=145.79.227.50:4374 , piece_id=Q72FSMVVPPCFCCIBIHAUUZUIQ3TB42XGA2AFBHYFESO3Z6HG2DEA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.208863379 (553h41m34.079946414s) [noise=true , address=145.79.227.46:4603 , piece_id=5OLCL5V2EBUBTOOAERF2UILITWPEPYK7HNYT6U7B7PRYAJ4PUYVA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.209441920 (553h41m34.080524965s) [noise=true , address=145.79.227.47:4266 , piece_id=6HQH62HPXMIKR4AXMNUWF74JBBN3756KYEFVLX5BSVJ5THE5TS5Q, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.213251424 (553h41m34.084334459s) [noise=true , address=145.79.227.8:4330 , piece_id=G6BYDKHAIKK7EPVXX2P7KNOFXTCICMPN53DAITHH4L6LRAV65EGA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.217652074 (553h41m34.088735109s) [noise=true , address=145.79.227.37:4566 , piece_id=UZCWMZHMXENHLHZXYVKRDWEHPIORV4FZRXYY3HDE2IAASQUXZVMA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.238730301 (553h41m34.109813336s) [noise=true , address=145.79.227.52:4446 , piece_id=AA2GMRF6XXJFHDQ3KMYEWE7CCQYBP4LWDWNPA2SMSMUXPHRMIHIQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.425277103 (553h41m34.296360148s) [noise=true , address=77.90.31.11:4554 , piece_id=DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.515389765 (553h41m34.3864728s) [noise=true , address=45.91.116.4:4937 , piece_id=DRIAIJZNCVK72UWNHN26J7OXJGGWBBGYFDB24SS36VI4XFOFQFPA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.516022371 (553h41m34.387105416s) [noise=true , address=45.91.116.8:4048 , piece_id=P37SE2DCIJSOETG66LEY2RXCIEE52UKOA2N5SPF27IOYITAOCBNQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:35.121095662 (553h41m34.992178707s) [noise=true , address=45.91.116.13:4155 , piece_id=IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:35.121121400 (553h41m34.992204435s) [seg_pos={0 0}, upload=485448]: Segment reached optimal threshold of 13 pieces. Dec 28 17:10:35.121316322 (553h41m34.992399367s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.121323455 (553h41m34.99240649s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.121641731 (553h41m34.992724766s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.121648545 (553h41m34.99273159s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.122210485 (553h41m34.99329353s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.122227136 (553h41m34.993310181s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.122252895 (553h41m34.99333593s) [seg_pos={0 0}, upload=485448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:10:35.122382432 (553h41m34.993465477s) [upload=485448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:10:35.122388914 (553h41m34.993471959s) [upload=485448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:10:35.452563433 (553h41m35.323646478s) [upload=485448]: Done uploading segments... Dec 28 17:10:35.452604561 (553h41m35.323687606s) [upload=485448]: Upload finished. err: <nil> Duration: 6.649279158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X........................X...............] duration: 28.658736831s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:11:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 28.658750346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX.] duration: 8.686059075s trace: https://jaeger.dev.storj.io/trace/7ee1234b4dd3f170 FAIL (2025-12-28 17:11:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ) context deadline exceeded ======LOGS====== Dec 28 17:10:31.923915099 (553h41m31.794998134s) [upload=485448]: Starting upload Dec 28 17:10:31.923984641 (553h41m31.795067676s) [upload=485448]: Done starting upload Dec 28 17:10:31.924099270 (553h41m31.795182315s) [upload=485448]: Uploading segments... Dec 28 17:10:31.924177739 (553h41m31.795260784s) [upload=485448]: Got next segment. Inline: false Dec 28 17:10:31.924213748 (553h41m31.795296783s) [upload=485448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:10:31.924218767 (553h41m31.795301802s) [upload=485448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:10:32.171366966 (553h41m32.042450011s) [seg_pos={0 0}, upload=485448]: Begin upload segment... Dec 28 17:10:32.171441368 (553h41m32.042524413s) [seg_pos={0 0}, upload=485448]: Done begin upload segment. Dec 28 17:10:32.171484040 (553h41m32.042567085s) [noise=true , address=145.79.227.50:4374 , piece_id=Q72FSMVVPPCFCCIBIHAUUZUIQ3TB42XGA2AFBHYFESO3Z6HG2DEA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.173952240 (553h41m32.045035275s) [noise=true , address=145.79.227.46:4603 , piece_id=5OLCL5V2EBUBTOOAERF2UILITWPEPYK7HNYT6U7B7PRYAJ4PUYVA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.174429098 (553h41m32.045512143s) [noise=true , address=45.91.116.4:4937 , piece_id=DRIAIJZNCVK72UWNHN26J7OXJGGWBBGYFDB24SS36VI4XFOFQFPA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.174724190 (553h41m32.045807225s) [noise=true , address=145.79.227.8:4330 , piece_id=G6BYDKHAIKK7EPVXX2P7KNOFXTCICMPN53DAITHH4L6LRAV65EGA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175039961 (553h41m32.046122996s) [noise=true , address=145.79.227.37:4566 , piece_id=UZCWMZHMXENHLHZXYVKRDWEHPIORV4FZRXYY3HDE2IAASQUXZVMA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175358577 (553h41m32.046441613s) [noise=true , address=145.79.227.39:4013 , piece_id=F55YK4E643BOGT7IVHIVBICZ5HLZCXIM2HHOUZUBYZ3V6ZIBETOA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175685741 (553h41m32.046768776s) [noise=true , address=145.79.227.38:4977 , piece_id=MTWFD4H5YVKN3G7NFQZ7TDLSSOFKWXCYRBB6ALYKSSRUCBEIHD2A, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.176729238 (553h41m32.047812273s) [noise=true , address=45.91.116.8:4048 , piece_id=P37SE2DCIJSOETG66LEY2RXCIEE52UKOA2N5SPF27IOYITAOCBNQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.176863583 (553h41m32.047946628s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.178256195 (553h41m32.04933923s) [noise=true , address=77.90.31.11:4554 , piece_id=DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.179005732 (553h41m32.050088768s) [noise=true , address=145.79.227.51:4411 , piece_id=QO3IXN3WINZNGI5XPKXJNG7EAGVBNHAPU2ADVCFMW3UN76MY4RBQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.182431747 (553h41m32.053514782s) [noise=true , address=45.91.116.13:4155 , piece_id=IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.185593568 (553h41m32.056676603s) [noise=true , address=145.79.227.52:4446 , piece_id=AA2GMRF6XXJFHDQ3KMYEWE7CCQYBP4LWDWNPA2SMSMUXPHRMIHIQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.186997140 (553h41m32.058080175s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.189407279 (553h41m32.060490354s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.194868729 (553h41m32.065951774s) [noise=true , address=145.79.227.47:4266 , piece_id=6HQH62HPXMIKR4AXMNUWF74JBBN3756KYEFVLX5BSVJ5THE5TS5Q, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:33.896978888 (553h41m33.768061963s) [upload=485448]: Next returned nil segment Dec 28 17:10:33.897003083 (553h41m33.768086129s) [upload=485448]: Waiting for error group managing segments... Dec 28 17:10:34.185240215 (553h41m34.05632326s) [noise=true , address=145.79.227.38:4977 , piece_id=MTWFD4H5YVKN3G7NFQZ7TDLSSOFKWXCYRBB6ALYKSSRUCBEIHD2A, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.190704710 (553h41m34.061787755s) [noise=true , address=145.79.227.51:4411 , piece_id=QO3IXN3WINZNGI5XPKXJNG7EAGVBNHAPU2ADVCFMW3UN76MY4RBQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.193934221 (553h41m34.065017256s) [noise=true , address=145.79.227.39:4013 , piece_id=F55YK4E643BOGT7IVHIVBICZ5HLZCXIM2HHOUZUBYZ3V6ZIBETOA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.194813936 (553h41m34.065896981s) [noise=true , address=145.79.227.50:4374 , piece_id=Q72FSMVVPPCFCCIBIHAUUZUIQ3TB42XGA2AFBHYFESO3Z6HG2DEA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.208863379 (553h41m34.079946414s) [noise=true , address=145.79.227.46:4603 , piece_id=5OLCL5V2EBUBTOOAERF2UILITWPEPYK7HNYT6U7B7PRYAJ4PUYVA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.209441920 (553h41m34.080524965s) [noise=true , address=145.79.227.47:4266 , piece_id=6HQH62HPXMIKR4AXMNUWF74JBBN3756KYEFVLX5BSVJ5THE5TS5Q, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.213251424 (553h41m34.084334459s) [noise=true , address=145.79.227.8:4330 , piece_id=G6BYDKHAIKK7EPVXX2P7KNOFXTCICMPN53DAITHH4L6LRAV65EGA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.217652074 (553h41m34.088735109s) [noise=true , address=145.79.227.37:4566 , piece_id=UZCWMZHMXENHLHZXYVKRDWEHPIORV4FZRXYY3HDE2IAASQUXZVMA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.238730301 (553h41m34.109813336s) [noise=true , address=145.79.227.52:4446 , piece_id=AA2GMRF6XXJFHDQ3KMYEWE7CCQYBP4LWDWNPA2SMSMUXPHRMIHIQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.425277103 (553h41m34.296360148s) [noise=true , address=77.90.31.11:4554 , piece_id=DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.515389765 (553h41m34.3864728s) [noise=true , address=45.91.116.4:4937 , piece_id=DRIAIJZNCVK72UWNHN26J7OXJGGWBBGYFDB24SS36VI4XFOFQFPA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.516022371 (553h41m34.387105416s) [noise=true , address=45.91.116.8:4048 , piece_id=P37SE2DCIJSOETG66LEY2RXCIEE52UKOA2N5SPF27IOYITAOCBNQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:35.121095662 (553h41m34.992178707s) [noise=true , address=45.91.116.13:4155 , piece_id=IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:35.121121400 (553h41m34.992204435s) [seg_pos={0 0}, upload=485448]: Segment reached optimal threshold of 13 pieces. Dec 28 17:10:35.121316322 (553h41m34.992399367s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.121323455 (553h41m34.99240649s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.121641731 (553h41m34.992724766s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.121648545 (553h41m34.99273159s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.122210485 (553h41m34.99329353s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.122227136 (553h41m34.993310181s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.122252895 (553h41m34.99333593s) [seg_pos={0 0}, upload=485448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:10:35.122382432 (553h41m34.993465477s) [upload=485448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:10:35.122388914 (553h41m34.993471959s) [upload=485448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:10:35.452563433 (553h41m35.323646478s) [upload=485448]: Done uploading segments... Dec 28 17:10:35.452604561 (553h41m35.323687606s) [upload=485448]: Upload finished. err: <nil> Duration: 8.686087879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X.........] duration: 3.381900385s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:11:28 UTC) orbiter warning: context deadline exceeded Duration: 3.381976251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X........................X..............] duration: 36.219271142s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:11:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 36.219288604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XXX] duration: 30.003446052s trace: https://jaeger.dev.storj.io/trace/7ee1234b4dd3f170 FAIL (2025-12-28 17:11:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ) context deadline exceeded ======LOGS====== Dec 28 17:10:31.923915099 (553h41m31.794998134s) [upload=485448]: Starting upload Dec 28 17:10:31.923984641 (553h41m31.795067676s) [upload=485448]: Done starting upload Dec 28 17:10:31.924099270 (553h41m31.795182315s) [upload=485448]: Uploading segments... Dec 28 17:10:31.924177739 (553h41m31.795260784s) [upload=485448]: Got next segment. Inline: false Dec 28 17:10:31.924213748 (553h41m31.795296783s) [upload=485448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:10:31.924218767 (553h41m31.795301802s) [upload=485448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:10:32.171366966 (553h41m32.042450011s) [seg_pos={0 0}, upload=485448]: Begin upload segment... Dec 28 17:10:32.171441368 (553h41m32.042524413s) [seg_pos={0 0}, upload=485448]: Done begin upload segment. Dec 28 17:10:32.171484040 (553h41m32.042567085s) [noise=true , address=145.79.227.50:4374 , piece_id=Q72FSMVVPPCFCCIBIHAUUZUIQ3TB42XGA2AFBHYFESO3Z6HG2DEA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.173952240 (553h41m32.045035275s) [noise=true , address=145.79.227.46:4603 , piece_id=5OLCL5V2EBUBTOOAERF2UILITWPEPYK7HNYT6U7B7PRYAJ4PUYVA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.174429098 (553h41m32.045512143s) [noise=true , address=45.91.116.4:4937 , piece_id=DRIAIJZNCVK72UWNHN26J7OXJGGWBBGYFDB24SS36VI4XFOFQFPA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.174724190 (553h41m32.045807225s) [noise=true , address=145.79.227.8:4330 , piece_id=G6BYDKHAIKK7EPVXX2P7KNOFXTCICMPN53DAITHH4L6LRAV65EGA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175039961 (553h41m32.046122996s) [noise=true , address=145.79.227.37:4566 , piece_id=UZCWMZHMXENHLHZXYVKRDWEHPIORV4FZRXYY3HDE2IAASQUXZVMA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175358577 (553h41m32.046441613s) [noise=true , address=145.79.227.39:4013 , piece_id=F55YK4E643BOGT7IVHIVBICZ5HLZCXIM2HHOUZUBYZ3V6ZIBETOA, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.175685741 (553h41m32.046768776s) [noise=true , address=145.79.227.38:4977 , piece_id=MTWFD4H5YVKN3G7NFQZ7TDLSSOFKWXCYRBB6ALYKSSRUCBEIHD2A, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.176729238 (553h41m32.047812273s) [noise=true , address=45.91.116.8:4048 , piece_id=P37SE2DCIJSOETG66LEY2RXCIEE52UKOA2N5SPF27IOYITAOCBNQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.176863583 (553h41m32.047946628s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.178256195 (553h41m32.04933923s) [noise=true , address=77.90.31.11:4554 , piece_id=DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.179005732 (553h41m32.050088768s) [noise=true , address=145.79.227.51:4411 , piece_id=QO3IXN3WINZNGI5XPKXJNG7EAGVBNHAPU2ADVCFMW3UN76MY4RBQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.182431747 (553h41m32.053514782s) [noise=true , address=45.91.116.13:4155 , piece_id=IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.185593568 (553h41m32.056676603s) [noise=true , address=145.79.227.52:4446 , piece_id=AA2GMRF6XXJFHDQ3KMYEWE7CCQYBP4LWDWNPA2SMSMUXPHRMIHIQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.186997140 (553h41m32.058080175s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.189407279 (553h41m32.060490354s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:32.194868729 (553h41m32.065951774s) [noise=true , address=145.79.227.47:4266 , piece_id=6HQH62HPXMIKR4AXMNUWF74JBBN3756KYEFVLX5BSVJ5THE5TS5Q, seg_pos={0 0}, upload=485448]: Uploading piece... Dec 28 17:10:33.896978888 (553h41m33.768061963s) [upload=485448]: Next returned nil segment Dec 28 17:10:33.897003083 (553h41m33.768086129s) [upload=485448]: Waiting for error group managing segments... Dec 28 17:10:34.185240215 (553h41m34.05632326s) [noise=true , address=145.79.227.38:4977 , piece_id=MTWFD4H5YVKN3G7NFQZ7TDLSSOFKWXCYRBB6ALYKSSRUCBEIHD2A, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.190704710 (553h41m34.061787755s) [noise=true , address=145.79.227.51:4411 , piece_id=QO3IXN3WINZNGI5XPKXJNG7EAGVBNHAPU2ADVCFMW3UN76MY4RBQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.193934221 (553h41m34.065017256s) [noise=true , address=145.79.227.39:4013 , piece_id=F55YK4E643BOGT7IVHIVBICZ5HLZCXIM2HHOUZUBYZ3V6ZIBETOA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.194813936 (553h41m34.065896981s) [noise=true , address=145.79.227.50:4374 , piece_id=Q72FSMVVPPCFCCIBIHAUUZUIQ3TB42XGA2AFBHYFESO3Z6HG2DEA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.208863379 (553h41m34.079946414s) [noise=true , address=145.79.227.46:4603 , piece_id=5OLCL5V2EBUBTOOAERF2UILITWPEPYK7HNYT6U7B7PRYAJ4PUYVA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.209441920 (553h41m34.080524965s) [noise=true , address=145.79.227.47:4266 , piece_id=6HQH62HPXMIKR4AXMNUWF74JBBN3756KYEFVLX5BSVJ5THE5TS5Q, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.213251424 (553h41m34.084334459s) [noise=true , address=145.79.227.8:4330 , piece_id=G6BYDKHAIKK7EPVXX2P7KNOFXTCICMPN53DAITHH4L6LRAV65EGA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.217652074 (553h41m34.088735109s) [noise=true , address=145.79.227.37:4566 , piece_id=UZCWMZHMXENHLHZXYVKRDWEHPIORV4FZRXYY3HDE2IAASQUXZVMA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.238730301 (553h41m34.109813336s) [noise=true , address=145.79.227.52:4446 , piece_id=AA2GMRF6XXJFHDQ3KMYEWE7CCQYBP4LWDWNPA2SMSMUXPHRMIHIQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.425277103 (553h41m34.296360148s) [noise=true , address=77.90.31.11:4554 , piece_id=DPJK7I6KBCM4SUVKNLIX32RPXQRR7ZTDEWIDT4DTJEHOPMWC6O2A, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.515389765 (553h41m34.3864728s) [noise=true , address=45.91.116.4:4937 , piece_id=DRIAIJZNCVK72UWNHN26J7OXJGGWBBGYFDB24SS36VI4XFOFQFPA, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:34.516022371 (553h41m34.387105416s) [noise=true , address=45.91.116.8:4048 , piece_id=P37SE2DCIJSOETG66LEY2RXCIEE52UKOA2N5SPF27IOYITAOCBNQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:35.121095662 (553h41m34.992178707s) [noise=true , address=45.91.116.13:4155 , piece_id=IBOXVSG3FXAK7L4OC4SU2SKLMSC665E3L4QAIAZ75SOXYAPJVQDQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: <nil> Dec 28 17:10:35.121121400 (553h41m34.992204435s) [seg_pos={0 0}, upload=485448]: Segment reached optimal threshold of 13 pieces. Dec 28 17:10:35.121316322 (553h41m34.992399367s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.121323455 (553h41m34.99240649s) [noise=true , address=45.91.116.11:4118 , piece_id=PVCWDCTRA7ZTSZE7457YSRB2C4RYZL6DPCZRCHQX5MNK6IQBNH7Q, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.121641731 (553h41m34.992724766s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.121648545 (553h41m34.99273159s) [noise=true , address=145.79.227.40:4050 , piece_id=QUX45O64RJKU7KQ25CAD7LJU4KSGFR2QBR34E2PBC2FFA7XGIEFQ, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.122210485 (553h41m34.99329353s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 17:10:35.122227136 (553h41m34.993310181s) [noise=true , address=145.79.227.29:4943 , piece_id=CFYYBD2YXMCEVAX76V5VNZGGICRS66Z4WM4HVIPB4ZQMEQ3CXSCQ, seg_pos={0 0}, upload=485448]: Long tail cancellation due to optimal threshold Dec 28 17:10:35.122252895 (553h41m34.99333593s) [seg_pos={0 0}, upload=485448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:10:35.122382432 (553h41m34.993465477s) [upload=485448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:10:35.122388914 (553h41m34.993471959s) [upload=485448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:10:35.452563433 (553h41m35.323646478s) [upload=485448]: Done uploading segments... Dec 28 17:10:35.452604561 (553h41m35.323687606s) [upload=485448]: Upload finished. err: <nil> Duration: 30.003469637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X........................X.............] duration: 47.987017695s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:10:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 47.987043866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X........] duration: 4.305061218s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:10:37 UTC) orbiter warning: context deadline exceeded Duration: 4.305090683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........XX] duration: 30.003672522s trace: https://jaeger.dev.storj.io/trace/776baeb80da8af99 FAIL (2025-12-28 17:10:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 17:09:31.790339679 (553h40m31.661422714s) [upload=485432]: Starting upload Dec 28 17:09:31.790401477 (553h40m31.661484512s) [upload=485432]: Done starting upload Dec 28 17:09:31.790553777 (553h40m31.661636812s) [upload=485432]: Uploading segments... Dec 28 17:09:31.790672713 (553h40m31.661755778s) [upload=485432]: Got next segment. Inline: false Dec 28 17:09:31.790722498 (553h40m31.661805543s) [upload=485432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:09:31.790729261 (553h40m31.661812306s) [upload=485432]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:09:32.038514324 (553h40m31.909597369s) [seg_pos={0 0}, upload=485432]: Begin upload segment... Dec 28 17:09:32.038557125 (553h40m31.90964017s) [seg_pos={0 0}, upload=485432]: Done begin upload segment. Dec 28 17:09:32.038583185 (553h40m31.90966622s) [noise=true , address=77.90.31.16:4736 , piece_id=X4BL76ZOFNJCDRI4RE5NOBXDXWU4AFIVALDMWAB5IOTBFOEEH2UQ, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.038692252 (553h40m31.909775287s) [noise=true , address=145.79.227.41:4086 , piece_id=MSDIYC6NY7OGEBSPUJ5S2UUEV6M3LAGGKC7CCVYXV5ISNQXI6K4A, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.039314738 (553h40m31.910397773s) [noise=true , address=145.79.227.51:4409 , piece_id=7ZMELTL3AVZGVXG3OYUKW2SW424V6WGVTS3S5V4QPZT6JZDD6W5A, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.039523155 (553h40m31.91060619s) [noise=true , address=145.79.227.39:4015 , piece_id=KDTNGMKYWZPJFAQLRGDHHYSYCSYVQB5WRE42KAMNYBZ5VQUCG7DA, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.039813046 (553h40m31.910896091s) [noise=true , address=45.91.116.10:4082 , piece_id=ZI4MYEBUXOOTXY7Q6SZPYRDUKT2SGNNGCD4GRIMCR67XITN2FTFA, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.040000644 (553h40m31.911083689s) [noise=true , address=145.79.227.16:4912 , piece_id=4635QEKS6KAHXRSAJQMS3GXKOB2LK7PAFF4LDHXYPXLBCXZPNXNA, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.040252473 (553h40m31.911335508s) [noise=true , address=45.91.116.5:4298 , piece_id=DLJVWVUBTMXEIHVUTVGMEGJEFO33EVLZA5SPS4SM5BKYHUOB6B6A, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.040396628 (553h40m31.911479663s) [noise=true , address=145.79.227.17:4972 , piece_id=7ACNDHPTOEVEEOUHAXWY7TU65JTTTH6D2XS6EBSUNW5D3YMNY4PQ, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.040726726 (553h40m31.911809761s) [noise=true , address=145.79.227.38:4978 , piece_id=D4U3MB5ZREVBGC6AV5DGYLXMZLMNWRMZRFS2ZSETJ3XSLY3NK5WA, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.040733870 (553h40m31.911816915s) [noise=true , address=145.79.227.50:4373 , piece_id=ICRRHVXOKPUSG2U5EW5RIASMKIMUDRWNR5VU7LBCKVB5PQFWL2UA, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.041067735 (553h40m31.91215077s) [noise=true , address=145.79.227.4:4185 , piece_id=ZHWS3KFSTT7IL7WMLH53AZ3OYXBJNJQDRFTHVHQMPV62HVUPXIMQ, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.041117410 (553h40m31.912200446s) [noise=true , address=145.79.227.5:4221 , piece_id=NGHV54B3FP6GVEVA65KSTKNQP4JEWN2LCFLSP3GENY3FWBX4XERQ, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.041492044 (553h40m31.912575089s) [noise=true , address=145.79.227.36:4517 , piece_id=BII7AF4DHBPY47TO2IO3C4W2ABM5Y34NXBGIDI2JF4UTWRG6AXOQ, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.041905791 (553h40m31.912988866s) [noise=true , address=77.90.31.13:4627 , piece_id=QRW54VJJFLNBAJKPNZ47HCPX6PKOMOAA7XQOFIC5YSO3XFMVIUGQ, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.042268152 (553h40m31.913351187s) [noise=true , address=145.79.227.6:4258 , piece_id=GVG2LBDNNNKJYZ4XQZEWJGEV3MUJI7DC6HSFS3YEWTWQIYJLT5EA, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:32.042492498 (553h40m31.913575543s) [noise=true , address=45.91.116.9:4262 , piece_id=37HGG47XBUZDXICT5IG4UDGTEEBLE3AP4ZFMCEVN4F3ERSPCDFZA, seg_pos={0 0}, upload=485432]: Uploading piece... Dec 28 17:09:33.659184162 (553h40m33.530267197s) [upload=485432]: Next returned nil segment Dec 28 17:09:33.659197017 (553h40m33.530280052s) [upload=485432]: Waiting for error group managing segments... Dec 28 17:09:34.040943178 (553h40m33.912026223s) [noise=true , address=145.79.227.38:4978 , piece_id=D4U3MB5ZREVBGC6AV5DGYLXMZLMNWRMZRFS2ZSETJ3XSLY3NK5WA, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.048548780 (553h40m33.919631815s) [noise=true , address=145.79.227.6:4258 , piece_id=GVG2LBDNNNKJYZ4XQZEWJGEV3MUJI7DC6HSFS3YEWTWQIYJLT5EA, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.053532932 (553h40m33.924615967s) [noise=true , address=145.79.227.39:4015 , piece_id=KDTNGMKYWZPJFAQLRGDHHYSYCSYVQB5WRE42KAMNYBZ5VQUCG7DA, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.056989944 (553h40m33.928072979s) [noise=true , address=145.79.227.4:4185 , piece_id=ZHWS3KFSTT7IL7WMLH53AZ3OYXBJNJQDRFTHVHQMPV62HVUPXIMQ, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.058772168 (553h40m33.929855213s) [noise=true , address=145.79.227.41:4086 , piece_id=MSDIYC6NY7OGEBSPUJ5S2UUEV6M3LAGGKC7CCVYXV5ISNQXI6K4A, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.058996024 (553h40m33.93007907s) [noise=true , address=145.79.227.51:4409 , piece_id=7ZMELTL3AVZGVXG3OYUKW2SW424V6WGVTS3S5V4QPZT6JZDD6W5A, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.070287977 (553h40m33.941371012s) [noise=true , address=145.79.227.5:4221 , piece_id=NGHV54B3FP6GVEVA65KSTKNQP4JEWN2LCFLSP3GENY3FWBX4XERQ, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.070503056 (553h40m33.941586101s) [noise=true , address=145.79.227.36:4517 , piece_id=BII7AF4DHBPY47TO2IO3C4W2ABM5Y34NXBGIDI2JF4UTWRG6AXOQ, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.071066178 (553h40m33.942149223s) [noise=true , address=145.79.227.50:4373 , piece_id=ICRRHVXOKPUSG2U5EW5RIASMKIMUDRWNR5VU7LBCKVB5PQFWL2UA, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.110318733 (553h40m33.981401768s) [noise=true , address=145.79.227.16:4912 , piece_id=4635QEKS6KAHXRSAJQMS3GXKOB2LK7PAFF4LDHXYPXLBCXZPNXNA, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.220018854 (553h40m34.091101889s) [noise=true , address=77.90.31.13:4627 , piece_id=QRW54VJJFLNBAJKPNZ47HCPX6PKOMOAA7XQOFIC5YSO3XFMVIUGQ, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.257695828 (553h40m34.128778873s) [noise=true , address=145.79.227.17:4972 , piece_id=7ACNDHPTOEVEEOUHAXWY7TU65JTTTH6D2XS6EBSUNW5D3YMNY4PQ, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.314777163 (553h40m34.185860198s) [noise=true , address=45.91.116.10:4082 , piece_id=ZI4MYEBUXOOTXY7Q6SZPYRDUKT2SGNNGCD4GRIMCR67XITN2FTFA, seg_pos={0 0}, upload=485432]: Done uploading piece. err: <nil> Dec 28 17:09:34.314835133 (553h40m34.185918178s) [seg_pos={0 0}, upload=485432]: Segment reached optimal threshold of 13 pieces. Dec 28 17:09:34.315080640 (553h40m34.186163675s) [noise=true , address=45.91.116.9:4262 , piece_id=37HGG47XBUZDXICT5IG4UDGTEEBLE3AP4ZFMCEVN4F3ERSPCDFZA, seg_pos={0 0}, upload=485432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 17:09:34.315091512 (553h40m34.186174547s) [noise=true , address=45.91.116.9:4262 , piece_id=37HGG47XBUZDXICT5IG4UDGTEEBLE3AP4ZFMCEVN4F3ERSPCDFZA, seg_pos={0 0}, upload=485432]: Long tail cancellation due to optimal threshold Dec 28 17:09:34.315147518 (553h40m34.186230553s) [noise=true , address=45.91.116.5:4298 , piece_id=DLJVWVUBTMXEIHVUTVGMEGJEFO33EVLZA5SPS4SM5BKYHUOB6B6A, seg_pos={0 0}, upload=485432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 17:09:34.315175141 (553h40m34.186258196s) [noise=true , address=45.91.116.5:4298 , piece_id=DLJVWVUBTMXEIHVUTVGMEGJEFO33EVLZA5SPS4SM5BKYHUOB6B6A, seg_pos={0 0}, upload=485432]: Long tail cancellation due to optimal threshold Dec 28 17:09:34.315252868 (553h40m34.186335923s) [noise=true , address=77.90.31.16:4736 , piece_id=X4BL76ZOFNJCDRI4RE5NOBXDXWU4AFIVALDMWAB5IOTBFOEEH2UQ, seg_pos={0 0}, upload=485432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 17:09:34.315264120 (553h40m34.186347155s) [noise=true , address=77.90.31.16:4736 , piece_id=X4BL76ZOFNJCDRI4RE5NOBXDXWU4AFIVALDMWAB5IOTBFOEEH2UQ, seg_pos={0 0}, upload=485432]: Long tail cancellation due to optimal threshold Dec 28 17:09:34.315284679 (553h40m34.186367714s) [seg_pos={0 0}, upload=485432]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:09:34.315363679 (553h40m34.186446714s) [upload=485432]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:09:34.315369721 (553h40m34.186452767s) [upload=485432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:09:34.611104018 (553h40m34.482187063s) [upload=485432]: Done uploading segments... Dec 28 17:09:34.611127623 (553h40m34.482210658s) [upload=485432]: Upload finished. err: <nil> Duration: 30.003699815s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X.......] duration: 3.543811436s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:09:53 UTC) orbiter warning: context deadline exceeded Duration: 3.543838457s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X......] duration: 3.635213074s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:09:19 UTC) orbiter warning: context deadline exceeded Duration: 3.635241027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X........................X............] duration: 18.981688593s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:09:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 18.981708431s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X.....] duration: 3.430750394s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:09:00 UTC) orbiter warning: context deadline exceeded Duration: 3.430784538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XX..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........X] duration: 30.003001766s trace: https://jaeger.dev.storj.io/trace/2dd3c53fb844b8db FAIL (2025-12-28 17:08:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 17:08:19.483394283 (553h39m19.354477318s) [upload=485415]: Starting upload Dec 28 17:08:19.483448928 (553h39m19.354531973s) [upload=485415]: Done starting upload Dec 28 17:08:19.483581569 (553h39m19.354664605s) [upload=485415]: Uploading segments... Dec 28 17:08:19.483675308 (553h39m19.354758343s) [upload=485415]: Got next segment. Inline: false Dec 28 17:08:19.483724182 (553h39m19.354807217s) [upload=485415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:08:19.483729161 (553h39m19.354812196s) [upload=485415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:08:19.730363633 (553h39m19.601446688s) [seg_pos={0 0}, upload=485415]: Begin upload segment... Dec 28 17:08:19.730554305 (553h39m19.601637351s) [seg_pos={0 0}, upload=485415]: Done begin upload segment. Dec 28 17:08:19.730629379 (553h39m19.601712454s) [noise=true , address=145.79.227.38:4977 , piece_id=XOUXYH2CQ6PK5WOUTEUCTWJZAQQ5T57EH2PRMQHFR25H6WXZ7GFA, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.730676148 (553h39m19.601759193s) [noise=true , address=145.79.227.4:4186 , piece_id=ZXZ2RVJ5IEFSXP3OFNE3WEKNWWOCNI7VYGOCNGKSSQ6CR4EYWG3Q, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.731476392 (553h39m19.602559427s) [noise=true , address=145.79.227.46:4601 , piece_id=C6J2BKFALFRUZKGOX4YARWTQY4MEG2USBZUNKDQZ7VI4K672B7TA, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.732003265 (553h39m19.60308631s) [noise=true , address=145.79.227.16:4910 , piece_id=7KX35KLNSEUJMPY5MSWMLU33XESLNHDJVB5MYWBTJYWPENS7T2EQ, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.732699801 (553h39m19.603782866s) [noise=true , address=145.79.227.30:4761 , piece_id=TKOTBEIHYDI5AF44UDAEJ2NBGFVKUFU2LNDETQL7ZSPQVHSTAXOA, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.733511016 (553h39m19.604594081s) [noise=true , address=145.79.227.11:4730 , piece_id=K7NEIYGUGJTRDH7EREPTVPS5LYKXRZ7XY2ZDSCTELUBKSTWK5AVQ, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.733774888 (553h39m19.604857954s) [noise=true , address=145.79.227.41:4086 , piece_id=6WH7KKHBYUE2L7Z6YWNGKJPZSJONXPP2KAC2UKENCZRVDNV5BBOA, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.734892346 (553h39m19.605975401s) [noise=true , address=45.91.116.7:4012 , piece_id=WF44WHITPCAUSP5LCQKIYN4G5TLQQPWBRTN6WFLAGTAVE47RA57A, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.735177699 (553h39m19.606260744s) [noise=true , address=145.79.227.50:4373 , piece_id=RQ3M56FHYFGRSIKPON6CUVC3NUB2RYMHHJCRLNSEYI6RTHAMETOQ, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.736435063 (553h39m19.607518119s) [noise=true , address=145.79.227.6:4257 , piece_id=CGUDCBQ37XJCTFLMOJVIJD5IVTV35IR6TL2FWT4W7NBJUZWMDI3A, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.736611991 (553h39m19.607695046s) [noise=true , address=45.91.116.9:4263 , piece_id=DERSNYOD6L7ARM3UFF5MP2Q7KHP4EBTDXEHTO574PXCCGYGNXT6A, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.737561188 (553h39m19.608644264s) [noise=true , address=145.79.227.7:4295 , piece_id=OUGTOQJ3XEB5BD44WDVFU4JETAO6MSNSTZMQPPE6DSKAGZ46WGIQ, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.737945150 (553h39m19.609028205s) [noise=true , address=45.91.116.6:4974 , piece_id=2W666DR2WWG3JWP4E4J77GNCOQFI42LD5GWMKEBHO6O3YJD5LQFA, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.738153356 (553h39m19.609236391s) [noise=true , address=145.79.227.9:4366 , piece_id=QIOFDXF7YP7PSGCNIOZT5KPHYPSZ65BYR25IGHRC4GDPGOKA5X2Q, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.738483825 (553h39m19.60956692s) [noise=true , address=77.90.31.12:4589 , piece_id=6JZIIKLZ7QCPEMHUVW7U4AN6QFPL2QS5CXTOEN5UZC4X44M7SKHA, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:19.738500407 (553h39m19.609583442s) [noise=true , address=145.79.227.43:4157 , piece_id=DY35HNYPB6TJEDJEODRZTTZYO75FK2622PLWKCL3VAJDD7OBTBMQ, seg_pos={0 0}, upload=485415]: Uploading piece... Dec 28 17:08:21.442262480 (553h39m21.313345515s) [upload=485415]: Next returned nil segment Dec 28 17:08:21.442483922 (553h39m21.313566977s) [upload=485415]: Waiting for error group managing segments... Dec 28 17:08:21.721540441 (553h39m21.592623476s) [noise=true , address=145.79.227.9:4366 , piece_id=QIOFDXF7YP7PSGCNIOZT5KPHYPSZ65BYR25IGHRC4GDPGOKA5X2Q, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.726217217 (553h39m21.597300262s) [noise=true , address=145.79.227.6:4257 , piece_id=CGUDCBQ37XJCTFLMOJVIJD5IVTV35IR6TL2FWT4W7NBJUZWMDI3A, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.735092828 (553h39m21.606175873s) [noise=true , address=145.79.227.11:4730 , piece_id=K7NEIYGUGJTRDH7EREPTVPS5LYKXRZ7XY2ZDSCTELUBKSTWK5AVQ, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.735903020 (553h39m21.606986055s) [noise=true , address=145.79.227.4:4186 , piece_id=ZXZ2RVJ5IEFSXP3OFNE3WEKNWWOCNI7VYGOCNGKSSQ6CR4EYWG3Q, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.738658759 (553h39m21.609741794s) [noise=true , address=145.79.227.41:4086 , piece_id=6WH7KKHBYUE2L7Z6YWNGKJPZSJONXPP2KAC2UKENCZRVDNV5BBOA, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.740033255 (553h39m21.61111629s) [noise=true , address=145.79.227.50:4373 , piece_id=RQ3M56FHYFGRSIKPON6CUVC3NUB2RYMHHJCRLNSEYI6RTHAMETOQ, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.745117647 (553h39m21.616200682s) [noise=true , address=145.79.227.7:4295 , piece_id=OUGTOQJ3XEB5BD44WDVFU4JETAO6MSNSTZMQPPE6DSKAGZ46WGIQ, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.746345324 (553h39m21.617428359s) [noise=true , address=145.79.227.46:4601 , piece_id=C6J2BKFALFRUZKGOX4YARWTQY4MEG2USBZUNKDQZ7VI4K672B7TA, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.747736624 (553h39m21.618819669s) [noise=true , address=145.79.227.16:4910 , piece_id=7KX35KLNSEUJMPY5MSWMLU33XESLNHDJVB5MYWBTJYWPENS7T2EQ, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.754498981 (553h39m21.625582016s) [noise=true , address=145.79.227.38:4977 , piece_id=XOUXYH2CQ6PK5WOUTEUCTWJZAQQ5T57EH2PRMQHFR25H6WXZ7GFA, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.984579320 (553h39m21.855662355s) [noise=true , address=45.91.116.9:4263 , piece_id=DERSNYOD6L7ARM3UFF5MP2Q7KHP4EBTDXEHTO574PXCCGYGNXT6A, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:21.985321463 (553h39m21.856404498s) [noise=true , address=45.91.116.7:4012 , piece_id=WF44WHITPCAUSP5LCQKIYN4G5TLQQPWBRTN6WFLAGTAVE47RA57A, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:22.017461464 (553h39m21.888544509s) [noise=true , address=77.90.31.12:4589 , piece_id=6JZIIKLZ7QCPEMHUVW7U4AN6QFPL2QS5CXTOEN5UZC4X44M7SKHA, seg_pos={0 0}, upload=485415]: Done uploading piece. err: <nil> Dec 28 17:08:22.017482424 (553h39m21.888565469s) [seg_pos={0 0}, upload=485415]: Segment reached optimal threshold of 13 pieces. Dec 28 17:08:22.017697744 (553h39m21.888780789s) [noise=true , address=45.91.116.6:4974 , piece_id=2W666DR2WWG3JWP4E4J77GNCOQFI42LD5GWMKEBHO6O3YJD5LQFA, seg_pos={0 0}, upload=485415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 17:08:22.017704677 (553h39m21.888787712s) [noise=true , address=45.91.116.6:4974 , piece_id=2W666DR2WWG3JWP4E4J77GNCOQFI42LD5GWMKEBHO6O3YJD5LQFA, seg_pos={0 0}, upload=485415]: Long tail cancellation due to optimal threshold Dec 28 17:08:22.018107705 (553h39m21.88919075s) [noise=true , address=145.79.227.30:4761 , piece_id=TKOTBEIHYDI5AF44UDAEJ2NBGFVKUFU2LNDETQL7ZSPQVHSTAXOA, seg_pos={0 0}, upload=485415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 17:08:22.018114788 (553h39m21.889197823s) [noise=true , address=145.79.227.30:4761 , piece_id=TKOTBEIHYDI5AF44UDAEJ2NBGFVKUFU2LNDETQL7ZSPQVHSTAXOA, seg_pos={0 0}, upload=485415]: Long tail cancellation due to optimal threshold Dec 28 17:08:22.018349325 (553h39m21.88943238s) [noise=true , address=145.79.227.43:4157 , piece_id=DY35HNYPB6TJEDJEODRZTTZYO75FK2622PLWKCL3VAJDD7OBTBMQ, seg_pos={0 0}, upload=485415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 28 17:08:22.018368792 (553h39m21.889451837s) [noise=true , address=145.79.227.43:4157 , piece_id=DY35HNYPB6TJEDJEODRZTTZYO75FK2622PLWKCL3VAJDD7OBTBMQ, seg_pos={0 0}, upload=485415]: Long tail cancellation due to optimal threshold Dec 28 17:08:22.018430499 (553h39m21.889513544s) [seg_pos={0 0}, upload=485415]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:08:22.018591747 (553h39m21.889674792s) [upload=485415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:08:22.018598850 (553h39m21.889681895s) [upload=485415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:08:22.307281359 (553h39m22.178364394s) [upload=485415]: Done uploading segments... Dec 28 17:08:22.307298492 (553h39m22.178381537s) [upload=485415]: Upload finished. err: <nil> Duration: 30.003024298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X........................X...........] duration: 37.910097789s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:08:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 37.910123387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..XX..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.........] duration: 6.093329513s trace: https://jaeger.dev.storj.io/trace/231f47b59a988108 FAIL (2025-12-28 17:08:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ) context deadline exceeded ======LOGS====== c 28 17:00:24.375346865 (553h31m24.24642991s) [upload=485298]: Starting upload Dec 28 17:00:24.375534311 (553h31m24.246617356s) [upload=485298]: Done starting upload Dec 28 17:00:24.375784819 (553h31m24.246867864s) [upload=485298]: Uploading segments... Dec 28 17:00:24.375874429 (553h31m24.246957494s) [upload=485298]: Got next segment. Inline: false Dec 28 17:00:24.375920276 (553h31m24.247003321s) [upload=485298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:00:24.375927601 (553h31m24.247010646s) [upload=485298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:00:24.619507796 (553h31m24.490590841s) [seg_pos={0 0}, upload=485298]: Begin upload segment... Dec 28 17:00:24.619570315 (553h31m24.490653361s) [seg_pos={0 0}, upload=485298]: Done begin upload segment. Dec 28 17:00:24.619612576 (553h31m24.490695621s) [noise=true , address=145.79.227.39:4014 , piece_id=ZII5RQGBGIUBWZUDU66YMKDXGMBBLBZCMWLAOEQ6EFTGXGRNEBUQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.620678015 (553h31m24.49176106s) [noise=true , address=145.79.227.42:4123 , piece_id=GZLWAPGEMSIXHJ2VX3FAZLIYGE4WAZLHLRZ2DYYXTOLOEIDTOKRQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.621412073 (553h31m24.492495118s) [noise=true , address=45.91.116.11:4117 , piece_id=A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.622256460 (553h31m24.493339505s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.622865540 (553h31m24.493948586s) [noise=true , address=145.79.227.26:4654 , piece_id=OYOFJEEDUVN7NADO64QXTAHWBM7XNZNSQIJWJSGVJN7SD2DXACCA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.623498015 (553h31m24.49458106s) [noise=true , address=145.79.227.32:4834 , piece_id=AE47IPPAPNWWR32344FRVLDPPRLUQHA3CAHWBXKVQV6CQH2OUJHA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.625645865 (553h31m24.49672892s) [noise=true , address=145.79.227.6:4259 , piece_id=B5I2KIRPCWXII5NZ656TC4Q7SEGYU2STZC7SIAIOG7HFIP7IHSIQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.627985159 (553h31m24.499068204s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.629899414 (553h31m24.500982459s) [noise=true , address=145.79.227.2:4114 , piece_id=TAF6S3EGNL7253NORVLZTRSLE77YFEK7O3ITE6WH5WSXWWKZGA5A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.630481843 (553h31m24.501564888s) [noise=true , address=145.79.227.37:4565 , piece_id=PWIKKB2WEGS63VOM3YGYFPQA23TXFXBCYGSEAQTQS2PT72G5CC4A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.631990426 (553h31m24.503073461s) [noise=true , address=77.90.31.10:4520 , piece_id=WMIAY666DLQHXYX5YFTPEAZCXVRDAYCK5UN3X4PDPL72NYJPODQQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.633691134 (553h31m24.504774179s) [noise=true , address=45.91.116.9:4261 , piece_id=B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.635322571 (553h31m24.506405616s) [noise=true , address=145.79.227.47:4267 , piece_id=V2XGHXRPUJEDDUOV43RIJXNDMSMU756HFFVYZQO2AP2NP2TESHFA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.636016282 (553h31m24.507099327s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.637649793 (553h31m24.508732838s) [noise=true , address=145.79.227.38:4977 , piece_id=3I7GFZV2TYMOEH3GJNMLUJZG3RFAUS6LVIL3QA5LMB7PWREC3NJA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.639410756 (553h31m24.510493801s) [noise=true , address=145.79.227.52:4446 , piece_id=TTKPUE4DPLNAM2KPDSSEWWKU375NJGZWYSI24VJOP6C4FDH5GICA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:26.246218791 (553h31m26.117301826s) [upload=485298]: Next returned nil segment Dec 28 17:00:26.246239028 (553h31m26.117322074s) [upload=485298]: Waiting for error group managing segments... Dec 28 17:00:26.626501094 (553h31m26.497584139s) [noise=true , address=145.79.227.37:4565 , piece_id=PWIKKB2WEGS63VOM3YGYFPQA23TXFXBCYGSEAQTQS2PT72G5CC4A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.648018187 (553h31m26.519101232s) [noise=true , address=145.79.227.42:4123 , piece_id=GZLWAPGEMSIXHJ2VX3FAZLIYGE4WAZLHLRZ2DYYXTOLOEIDTOKRQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.653471531 (553h31m26.524554576s) [noise=true , address=145.79.227.26:4654 , piece_id=OYOFJEEDUVN7NADO64QXTAHWBM7XNZNSQIJWJSGVJN7SD2DXACCA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.654358009 (553h31m26.525441064s) [noise=true , address=145.79.227.52:4446 , piece_id=TTKPUE4DPLNAM2KPDSSEWWKU375NJGZWYSI24VJOP6C4FDH5GICA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.658975371 (553h31m26.530058416s) [noise=true , address=145.79.227.38:4977 , piece_id=3I7GFZV2TYMOEH3GJNMLUJZG3RFAUS6LVIL3QA5LMB7PWREC3NJA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.659266436 (553h31m26.530349481s) [noise=true , address=145.79.227.6:4259 , piece_id=B5I2KIRPCWXII5NZ656TC4Q7SEGYU2STZC7SIAIOG7HFIP7IHSIQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.668434423 (553h31m26.539517468s) [noise=true , address=145.79.227.32:4834 , piece_id=AE47IPPAPNWWR32344FRVLDPPRLUQHA3CAHWBXKVQV6CQH2OUJHA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.670782375 (553h31m26.54186542s) [noise=true , address=145.79.227.2:4114 , piece_id=TAF6S3EGNL7253NORVLZTRSLE77YFEK7O3ITE6WH5WSXWWKZGA5A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.676646641 (553h31m26.547729686s) [noise=true , address=145.79.227.47:4267 , piece_id=V2XGHXRPUJEDDUOV43RIJXNDMSMU756HFFVYZQO2AP2NP2TESHFA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.682458929 (553h31m26.553541984s) [noise=true , address=145.79.227.39:4014 , piece_id=ZII5RQGBGIUBWZUDU66YMKDXGMBBLBZCMWLAOEQ6EFTGXGRNEBUQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.846734723 (553h31m26.717817758s) [noise=true , address=45.91.116.11:4117 , piece_id=A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.930023020 (553h31m26.801106075s) [noise=true , address=77.90.31.10:4520 , piece_id=WMIAY666DLQHXYX5YFTPEAZCXVRDAYCK5UN3X4PDPL72NYJPODQQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.984810186 (553h31m26.855893231s) [noise=true , address=45.91.116.9:4261 , piece_id=B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.984840824 (553h31m26.855923869s) [seg_pos={0 0}, upload=485298]: Segment reached optimal threshold of 13 pieces. Dec 28 17:00:26.985134354 (553h31m26.856217399s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 17:00:26.985144422 (553h31m26.856227468s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985451287 (553h31m26.856534332s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 17:00:26.985461847 (553h31m26.856544892s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985661427 (553h31m26.856744482s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection Dec 28 17:00:26.985671556 (553h31m26.856754611s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985696233 (553h31m26.856779278s) [seg_pos={0 0}, upload=485298]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:00:26.985788600 (553h31m26.856871645s) [upload=485298]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:00:26.985851028 (553h31m26.856934073s) [upload=485298]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:00:27.278886336 (553h31m27.149969381s) [upload=485298]: Done uploading segments... Dec 28 17:00:27.278911774 (553h31m27.149994819s) [upload=485298]: Upload finished. err: <nil> Duration: 6.093360432s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X....] duration: 3.502171238s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:07:53 UTC) orbiter warning: context deadline exceeded Duration: 3.502202478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..XX..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX........] duration: 6.478627773s trace: https://jaeger.dev.storj.io/trace/231f47b59a988108 FAIL (2025-12-28 17:07:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ) context deadline exceeded ======LOGS====== c 28 17:00:24.375346865 (553h31m24.24642991s) [upload=485298]: Starting upload Dec 28 17:00:24.375534311 (553h31m24.246617356s) [upload=485298]: Done starting upload Dec 28 17:00:24.375784819 (553h31m24.246867864s) [upload=485298]: Uploading segments... Dec 28 17:00:24.375874429 (553h31m24.246957494s) [upload=485298]: Got next segment. Inline: false Dec 28 17:00:24.375920276 (553h31m24.247003321s) [upload=485298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:00:24.375927601 (553h31m24.247010646s) [upload=485298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:00:24.619507796 (553h31m24.490590841s) [seg_pos={0 0}, upload=485298]: Begin upload segment... Dec 28 17:00:24.619570315 (553h31m24.490653361s) [seg_pos={0 0}, upload=485298]: Done begin upload segment. Dec 28 17:00:24.619612576 (553h31m24.490695621s) [noise=true , address=145.79.227.39:4014 , piece_id=ZII5RQGBGIUBWZUDU66YMKDXGMBBLBZCMWLAOEQ6EFTGXGRNEBUQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.620678015 (553h31m24.49176106s) [noise=true , address=145.79.227.42:4123 , piece_id=GZLWAPGEMSIXHJ2VX3FAZLIYGE4WAZLHLRZ2DYYXTOLOEIDTOKRQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.621412073 (553h31m24.492495118s) [noise=true , address=45.91.116.11:4117 , piece_id=A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.622256460 (553h31m24.493339505s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.622865540 (553h31m24.493948586s) [noise=true , address=145.79.227.26:4654 , piece_id=OYOFJEEDUVN7NADO64QXTAHWBM7XNZNSQIJWJSGVJN7SD2DXACCA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.623498015 (553h31m24.49458106s) [noise=true , address=145.79.227.32:4834 , piece_id=AE47IPPAPNWWR32344FRVLDPPRLUQHA3CAHWBXKVQV6CQH2OUJHA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.625645865 (553h31m24.49672892s) [noise=true , address=145.79.227.6:4259 , piece_id=B5I2KIRPCWXII5NZ656TC4Q7SEGYU2STZC7SIAIOG7HFIP7IHSIQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.627985159 (553h31m24.499068204s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.629899414 (553h31m24.500982459s) [noise=true , address=145.79.227.2:4114 , piece_id=TAF6S3EGNL7253NORVLZTRSLE77YFEK7O3ITE6WH5WSXWWKZGA5A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.630481843 (553h31m24.501564888s) [noise=true , address=145.79.227.37:4565 , piece_id=PWIKKB2WEGS63VOM3YGYFPQA23TXFXBCYGSEAQTQS2PT72G5CC4A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.631990426 (553h31m24.503073461s) [noise=true , address=77.90.31.10:4520 , piece_id=WMIAY666DLQHXYX5YFTPEAZCXVRDAYCK5UN3X4PDPL72NYJPODQQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.633691134 (553h31m24.504774179s) [noise=true , address=45.91.116.9:4261 , piece_id=B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.635322571 (553h31m24.506405616s) [noise=true , address=145.79.227.47:4267 , piece_id=V2XGHXRPUJEDDUOV43RIJXNDMSMU756HFFVYZQO2AP2NP2TESHFA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.636016282 (553h31m24.507099327s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.637649793 (553h31m24.508732838s) [noise=true , address=145.79.227.38:4977 , piece_id=3I7GFZV2TYMOEH3GJNMLUJZG3RFAUS6LVIL3QA5LMB7PWREC3NJA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.639410756 (553h31m24.510493801s) [noise=true , address=145.79.227.52:4446 , piece_id=TTKPUE4DPLNAM2KPDSSEWWKU375NJGZWYSI24VJOP6C4FDH5GICA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:26.246218791 (553h31m26.117301826s) [upload=485298]: Next returned nil segment Dec 28 17:00:26.246239028 (553h31m26.117322074s) [upload=485298]: Waiting for error group managing segments... Dec 28 17:00:26.626501094 (553h31m26.497584139s) [noise=true , address=145.79.227.37:4565 , piece_id=PWIKKB2WEGS63VOM3YGYFPQA23TXFXBCYGSEAQTQS2PT72G5CC4A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.648018187 (553h31m26.519101232s) [noise=true , address=145.79.227.42:4123 , piece_id=GZLWAPGEMSIXHJ2VX3FAZLIYGE4WAZLHLRZ2DYYXTOLOEIDTOKRQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.653471531 (553h31m26.524554576s) [noise=true , address=145.79.227.26:4654 , piece_id=OYOFJEEDUVN7NADO64QXTAHWBM7XNZNSQIJWJSGVJN7SD2DXACCA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.654358009 (553h31m26.525441064s) [noise=true , address=145.79.227.52:4446 , piece_id=TTKPUE4DPLNAM2KPDSSEWWKU375NJGZWYSI24VJOP6C4FDH5GICA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.658975371 (553h31m26.530058416s) [noise=true , address=145.79.227.38:4977 , piece_id=3I7GFZV2TYMOEH3GJNMLUJZG3RFAUS6LVIL3QA5LMB7PWREC3NJA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.659266436 (553h31m26.530349481s) [noise=true , address=145.79.227.6:4259 , piece_id=B5I2KIRPCWXII5NZ656TC4Q7SEGYU2STZC7SIAIOG7HFIP7IHSIQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.668434423 (553h31m26.539517468s) [noise=true , address=145.79.227.32:4834 , piece_id=AE47IPPAPNWWR32344FRVLDPPRLUQHA3CAHWBXKVQV6CQH2OUJHA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.670782375 (553h31m26.54186542s) [noise=true , address=145.79.227.2:4114 , piece_id=TAF6S3EGNL7253NORVLZTRSLE77YFEK7O3ITE6WH5WSXWWKZGA5A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.676646641 (553h31m26.547729686s) [noise=true , address=145.79.227.47:4267 , piece_id=V2XGHXRPUJEDDUOV43RIJXNDMSMU756HFFVYZQO2AP2NP2TESHFA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.682458929 (553h31m26.553541984s) [noise=true , address=145.79.227.39:4014 , piece_id=ZII5RQGBGIUBWZUDU66YMKDXGMBBLBZCMWLAOEQ6EFTGXGRNEBUQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.846734723 (553h31m26.717817758s) [noise=true , address=45.91.116.11:4117 , piece_id=A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.930023020 (553h31m26.801106075s) [noise=true , address=77.90.31.10:4520 , piece_id=WMIAY666DLQHXYX5YFTPEAZCXVRDAYCK5UN3X4PDPL72NYJPODQQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.984810186 (553h31m26.855893231s) [noise=true , address=45.91.116.9:4261 , piece_id=B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.984840824 (553h31m26.855923869s) [seg_pos={0 0}, upload=485298]: Segment reached optimal threshold of 13 pieces. Dec 28 17:00:26.985134354 (553h31m26.856217399s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 17:00:26.985144422 (553h31m26.856227468s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985451287 (553h31m26.856534332s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 17:00:26.985461847 (553h31m26.856544892s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985661427 (553h31m26.856744482s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection Dec 28 17:00:26.985671556 (553h31m26.856754611s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985696233 (553h31m26.856779278s) [seg_pos={0 0}, upload=485298]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:00:26.985788600 (553h31m26.856871645s) [upload=485298]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:00:26.985851028 (553h31m26.856934073s) [upload=485298]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:00:27.278886336 (553h31m27.149969381s) [upload=485298]: Done uploading segments... Dec 28 17:00:27.278911774 (553h31m27.149994819s) [upload=485298]: Upload finished. err: <nil> Duration: 6.478669483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X........................X..........] duration: 15.448403485s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:06:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 15.448431639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X...] duration: 3.64410588s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:06:39 UTC) orbiter warning: context deadline exceeded Duration: 3.644129635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X..XX..XXXX...X...XX....X.X.XXX..X.X...XX...XX..XX.......] duration: 7.590823329s trace: https://jaeger.dev.storj.io/trace/231f47b59a988108 FAIL (2025-12-28 17:06:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ) context deadline exceeded ======LOGS====== c 28 17:00:24.375346865 (553h31m24.24642991s) [upload=485298]: Starting upload Dec 28 17:00:24.375534311 (553h31m24.246617356s) [upload=485298]: Done starting upload Dec 28 17:00:24.375784819 (553h31m24.246867864s) [upload=485298]: Uploading segments... Dec 28 17:00:24.375874429 (553h31m24.246957494s) [upload=485298]: Got next segment. Inline: false Dec 28 17:00:24.375920276 (553h31m24.247003321s) [upload=485298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:00:24.375927601 (553h31m24.247010646s) [upload=485298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:00:24.619507796 (553h31m24.490590841s) [seg_pos={0 0}, upload=485298]: Begin upload segment... Dec 28 17:00:24.619570315 (553h31m24.490653361s) [seg_pos={0 0}, upload=485298]: Done begin upload segment. Dec 28 17:00:24.619612576 (553h31m24.490695621s) [noise=true , address=145.79.227.39:4014 , piece_id=ZII5RQGBGIUBWZUDU66YMKDXGMBBLBZCMWLAOEQ6EFTGXGRNEBUQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.620678015 (553h31m24.49176106s) [noise=true , address=145.79.227.42:4123 , piece_id=GZLWAPGEMSIXHJ2VX3FAZLIYGE4WAZLHLRZ2DYYXTOLOEIDTOKRQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.621412073 (553h31m24.492495118s) [noise=true , address=45.91.116.11:4117 , piece_id=A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.622256460 (553h31m24.493339505s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.622865540 (553h31m24.493948586s) [noise=true , address=145.79.227.26:4654 , piece_id=OYOFJEEDUVN7NADO64QXTAHWBM7XNZNSQIJWJSGVJN7SD2DXACCA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.623498015 (553h31m24.49458106s) [noise=true , address=145.79.227.32:4834 , piece_id=AE47IPPAPNWWR32344FRVLDPPRLUQHA3CAHWBXKVQV6CQH2OUJHA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.625645865 (553h31m24.49672892s) [noise=true , address=145.79.227.6:4259 , piece_id=B5I2KIRPCWXII5NZ656TC4Q7SEGYU2STZC7SIAIOG7HFIP7IHSIQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.627985159 (553h31m24.499068204s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.629899414 (553h31m24.500982459s) [noise=true , address=145.79.227.2:4114 , piece_id=TAF6S3EGNL7253NORVLZTRSLE77YFEK7O3ITE6WH5WSXWWKZGA5A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.630481843 (553h31m24.501564888s) [noise=true , address=145.79.227.37:4565 , piece_id=PWIKKB2WEGS63VOM3YGYFPQA23TXFXBCYGSEAQTQS2PT72G5CC4A, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.631990426 (553h31m24.503073461s) [noise=true , address=77.90.31.10:4520 , piece_id=WMIAY666DLQHXYX5YFTPEAZCXVRDAYCK5UN3X4PDPL72NYJPODQQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.633691134 (553h31m24.504774179s) [noise=true , address=45.91.116.9:4261 , piece_id=B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.635322571 (553h31m24.506405616s) [noise=true , address=145.79.227.47:4267 , piece_id=V2XGHXRPUJEDDUOV43RIJXNDMSMU756HFFVYZQO2AP2NP2TESHFA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.636016282 (553h31m24.507099327s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.637649793 (553h31m24.508732838s) [noise=true , address=145.79.227.38:4977 , piece_id=3I7GFZV2TYMOEH3GJNMLUJZG3RFAUS6LVIL3QA5LMB7PWREC3NJA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:24.639410756 (553h31m24.510493801s) [noise=true , address=145.79.227.52:4446 , piece_id=TTKPUE4DPLNAM2KPDSSEWWKU375NJGZWYSI24VJOP6C4FDH5GICA, seg_pos={0 0}, upload=485298]: Uploading piece... Dec 28 17:00:26.246218791 (553h31m26.117301826s) [upload=485298]: Next returned nil segment Dec 28 17:00:26.246239028 (553h31m26.117322074s) [upload=485298]: Waiting for error group managing segments... Dec 28 17:00:26.626501094 (553h31m26.497584139s) [noise=true , address=145.79.227.37:4565 , piece_id=PWIKKB2WEGS63VOM3YGYFPQA23TXFXBCYGSEAQTQS2PT72G5CC4A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.648018187 (553h31m26.519101232s) [noise=true , address=145.79.227.42:4123 , piece_id=GZLWAPGEMSIXHJ2VX3FAZLIYGE4WAZLHLRZ2DYYXTOLOEIDTOKRQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.653471531 (553h31m26.524554576s) [noise=true , address=145.79.227.26:4654 , piece_id=OYOFJEEDUVN7NADO64QXTAHWBM7XNZNSQIJWJSGVJN7SD2DXACCA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.654358009 (553h31m26.525441064s) [noise=true , address=145.79.227.52:4446 , piece_id=TTKPUE4DPLNAM2KPDSSEWWKU375NJGZWYSI24VJOP6C4FDH5GICA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.658975371 (553h31m26.530058416s) [noise=true , address=145.79.227.38:4977 , piece_id=3I7GFZV2TYMOEH3GJNMLUJZG3RFAUS6LVIL3QA5LMB7PWREC3NJA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.659266436 (553h31m26.530349481s) [noise=true , address=145.79.227.6:4259 , piece_id=B5I2KIRPCWXII5NZ656TC4Q7SEGYU2STZC7SIAIOG7HFIP7IHSIQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.668434423 (553h31m26.539517468s) [noise=true , address=145.79.227.32:4834 , piece_id=AE47IPPAPNWWR32344FRVLDPPRLUQHA3CAHWBXKVQV6CQH2OUJHA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.670782375 (553h31m26.54186542s) [noise=true , address=145.79.227.2:4114 , piece_id=TAF6S3EGNL7253NORVLZTRSLE77YFEK7O3ITE6WH5WSXWWKZGA5A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.676646641 (553h31m26.547729686s) [noise=true , address=145.79.227.47:4267 , piece_id=V2XGHXRPUJEDDUOV43RIJXNDMSMU756HFFVYZQO2AP2NP2TESHFA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.682458929 (553h31m26.553541984s) [noise=true , address=145.79.227.39:4014 , piece_id=ZII5RQGBGIUBWZUDU66YMKDXGMBBLBZCMWLAOEQ6EFTGXGRNEBUQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.846734723 (553h31m26.717817758s) [noise=true , address=45.91.116.11:4117 , piece_id=A5UPP2NGZASYBANP5BC5UWYXRLP4ELQGHUECCIESBHX3LCRZ6AWA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.930023020 (553h31m26.801106075s) [noise=true , address=77.90.31.10:4520 , piece_id=WMIAY666DLQHXYX5YFTPEAZCXVRDAYCK5UN3X4PDPL72NYJPODQQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.984810186 (553h31m26.855893231s) [noise=true , address=45.91.116.9:4261 , piece_id=B3OOCBNY2SERMBO6A56PTL4VFVTRNCF4NGZ3V2WGXZ2L7FAXMFZQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: <nil> Dec 28 17:00:26.984840824 (553h31m26.855923869s) [seg_pos={0 0}, upload=485298]: Segment reached optimal threshold of 13 pieces. Dec 28 17:00:26.985134354 (553h31m26.856217399s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 17:00:26.985144422 (553h31m26.856227468s) [noise=true , address=45.91.116.12:4226 , piece_id=JPDLCORHON6YHBMM3NDZ4OTBPPB6LTBHZLBCKWQQGGLLHAILFYLA, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985451287 (553h31m26.856534332s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 17:00:26.985461847 (553h31m26.856544892s) [noise=true , address=45.91.116.8:4045 , piece_id=N7BBGRXNYC4OSJ4247THZHNDHRY5AICIPGNPR34LZVEI5YJYUU4A, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985661427 (553h31m26.856744482s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41054->77.90.31.11:4553: use of closed network connection Dec 28 17:00:26.985671556 (553h31m26.856754611s) [noise=true , address=77.90.31.11:4553 , piece_id=FYO4ZABOZKLO635PTFAKVZDLL5TUIZA44P2WZJZ3BOATNAEWMIYQ, seg_pos={0 0}, upload=485298]: Long tail cancellation due to optimal threshold Dec 28 17:00:26.985696233 (553h31m26.856779278s) [seg_pos={0 0}, upload=485298]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:00:26.985788600 (553h31m26.856871645s) [upload=485298]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:00:26.985851028 (553h31m26.856934073s) [upload=485298]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:00:27.278886336 (553h31m27.149969381s) [upload=485298]: Done uploading segments... Dec 28 17:00:27.278911774 (553h31m27.149994819s) [upload=485298]: Upload finished. err: <nil> Duration: 7.59084463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X........................X.........] duration: 37.738587912s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:06:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 37.7386178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.......X..X..] duration: 4.435513706s trace: https://jaeger.dev.storj.io/trace/315400ed124647e5 FAIL (2025-12-28 17:05:38 UTC) orbiter warning: context deadline exceeded Duration: 4.435539304s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)