Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.....XX......X................XX.....................] duration: 23.108022812s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:05:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 23.10804258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X...X......X.X..X..XXX...........X.......X..X.XX..X..X....XX] duration: 30.003310106s trace: https://jaeger.dev.storj.io/trace/47d3fa56a87bc67f FAIL (2025-12-27 16:04:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Dec 27 16:04:28.542786609 (528h35m28.413869654s) [upload=463619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:04:28.542791348 (528h35m28.413874383s) [upload=463619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:04:28.789365140 (528h35m28.660448225s) [seg_pos={0 0}, upload=463619]: Begin upload segment... Dec 27 16:04:28.789432538 (528h35m28.660515583s) [seg_pos={0 0}, upload=463619]: Done begin upload segment. Dec 27 16:04:28.789466082 (528h35m28.660549127s) [noise=true , address=77.90.31.10:4520 , piece_id=6SEIKD53W4W6GM6IM6MPBWG5WYFCRLPTIRWAARLK3PSHDX3IABJQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.789474397 (528h35m28.660557443s) [noise=true , address=145.79.227.48:4302 , piece_id=IARJWR2J54O26SXKMARVGNAXGFIXHFH3PFV2TV3J4UCJ3AIAISNQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.789671513 (528h35m28.660754558s) [noise=true , address=77.90.31.19:4841 , piece_id=HHKXHVITGSKYXKABTYMSFHJNRJDDHXOR3ONXIHI2KHBEMLTXAU7Q, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.790105660 (528h35m28.661188695s) [noise=true , address=77.90.31.11:4555 , piece_id=HE2WOM2PJXDCJBYLK43JKC2TTBFCZR4VGRCGJVEMJJN4XP2MUPLA, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.790365214 (528h35m28.661448259s) [noise=true , address=145.79.227.39:4015 , piece_id=X7CYQ7UYFYT7CXYO5E2WCXEXSVO3PVETDQRQ6PDP4T44I6PBAWYQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.790719158 (528h35m28.661802193s) [noise=true , address=145.79.227.4:4187 , piece_id=E4OFHFI4EM7ZUQOXGUGSJN77PKT2EAXF67AQX37VMZCN6RTTDMRA, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.791100664 (528h35m28.66218371s) [noise=true , address=45.91.116.6:4973 , piece_id=IRXLUPXPNW3DF3NXQG6BJPCUWFW3P4IBRCPCCGU54LTZC6W7R2GQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.791420884 (528h35m28.662503919s) [noise=true , address=145.79.227.7:4295 , piece_id=KSO2FXX3H2FSXGY44HGW27ZWSOJT6UL7RVGHWKMKQYTKYFV74YWQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.791755472 (528h35m28.662838517s) [noise=true , address=145.79.227.38:4978 , piece_id=BB2PYZDPPJ3IW5AIGKAZ4KR6TINYARVWJ4IC6ZQI2N3U2E6T2ECA, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.792108714 (528h35m28.663191749s) [noise=true , address=45.91.116.12:4226 , piece_id=JZ7NE56XB5XWUAOZHI3BCA5KXGP4PJKTHWURSXXZD6HES43I6RFA, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.792447890 (528h35m28.663530925s) [noise=true , address=45.91.116.14:4192 , piece_id=WT2J63BZM4CUHYCTMBXYSFIXGQ6GEOQPM5IDN6O6YQ2ISAMLXY2Q, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.792777217 (528h35m28.663860252s) [noise=true , address=145.79.227.16:4910 , piece_id=ITHXHQPHZ5RJC27FGHBHNRJHXS4FMFOUHKK5N6OIMIPUVPM73CYQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.794217851 (528h35m28.665300896s) [noise=true , address=77.90.31.14:4661 , piece_id=W6OKF2H5ITRYBUNGXL7LAKHJ33MFKGY5ANJL5ADDSQTL6SCCZIEQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.794571073 (528h35m28.665654108s) [noise=true , address=145.79.227.46:4603 , piece_id=W73WWIICKYW5XW7DT5VPJIKX3VQJES55MV4LOSUOKKARRFCHZUFQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.795286896 (528h35m28.666369941s) [noise=true , address=145.79.227.43:4158 , piece_id=IDIGTWO3ZLOEDBBOERJLJOJ6CBMPDY6Z4PUCGLJDFT5YMXKK5PIQ, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:28.796589186 (528h35m28.667672221s) [noise=true , address=45.91.116.5:4299 , piece_id=QJEDEFZRKSADB7HFUTQQ27VDOEJZX5N6EFBJTJRTIW6TNAXHTZEA, seg_pos={0 0}, upload=463619]: Uploading piece... Dec 27 16:04:30.494880817 (528h35m30.365963862s) [upload=463619]: Next returned nil segment Dec 27 16:04:30.494899483 (528h35m30.365982528s) [upload=463619]: Waiting for error group managing segments... Dec 27 16:04:30.795698989 (528h35m30.666782034s) [noise=true , address=145.79.227.39:4015 , piece_id=X7CYQ7UYFYT7CXYO5E2WCXEXSVO3PVETDQRQ6PDP4T44I6PBAWYQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:30.797760314 (528h35m30.668843359s) [noise=true , address=145.79.227.38:4978 , piece_id=BB2PYZDPPJ3IW5AIGKAZ4KR6TINYARVWJ4IC6ZQI2N3U2E6T2ECA, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:30.803641413 (528h35m30.674724448s) [noise=true , address=145.79.227.4:4187 , piece_id=E4OFHFI4EM7ZUQOXGUGSJN77PKT2EAXF67AQX37VMZCN6RTTDMRA, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:30.810804823 (528h35m30.681887888s) [noise=true , address=145.79.227.46:4603 , piece_id=W73WWIICKYW5XW7DT5VPJIKX3VQJES55MV4LOSUOKKARRFCHZUFQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:30.814765304 (528h35m30.685848359s) [noise=true , address=145.79.227.48:4302 , piece_id=IARJWR2J54O26SXKMARVGNAXGFIXHFH3PFV2TV3J4UCJ3AIAISNQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:30.817400743 (528h35m30.688483788s) [noise=true , address=145.79.227.16:4910 , piece_id=ITHXHQPHZ5RJC27FGHBHNRJHXS4FMFOUHKK5N6OIMIPUVPM73CYQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:30.822824150 (528h35m30.693907195s) [noise=true , address=145.79.227.7:4295 , piece_id=KSO2FXX3H2FSXGY44HGW27ZWSOJT6UL7RVGHWKMKQYTKYFV74YWQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:30.838614618 (528h35m30.709697653s) [noise=true , address=145.79.227.43:4158 , piece_id=IDIGTWO3ZLOEDBBOERJLJOJ6CBMPDY6Z4PUCGLJDFT5YMXKK5PIQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:31.022807153 (528h35m30.893890188s) [noise=true , address=77.90.31.14:4661 , piece_id=W6OKF2H5ITRYBUNGXL7LAKHJ33MFKGY5ANJL5ADDSQTL6SCCZIEQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:31.042778502 (528h35m30.913861537s) [noise=true , address=45.91.116.12:4226 , piece_id=JZ7NE56XB5XWUAOZHI3BCA5KXGP4PJKTHWURSXXZD6HES43I6RFA, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:31.121893952 (528h35m30.992976997s) [noise=true , address=45.91.116.14:4192 , piece_id=WT2J63BZM4CUHYCTMBXYSFIXGQ6GEOQPM5IDN6O6YQ2ISAMLXY2Q, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:31.151402943 (528h35m31.022485988s) [noise=true , address=45.91.116.5:4299 , piece_id=QJEDEFZRKSADB7HFUTQQ27VDOEJZX5N6EFBJTJRTIW6TNAXHTZEA, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:31.183731863 (528h35m31.054814898s) [noise=true , address=45.91.116.6:4973 , piece_id=IRXLUPXPNW3DF3NXQG6BJPCUWFW3P4IBRCPCCGU54LTZC6W7R2GQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: <nil> Dec 27 16:04:31.183753023 (528h35m31.054836058s) [seg_pos={0 0}, upload=463619]: Segment reached optimal threshold of 13 pieces. Dec 27 16:04:31.183944247 (528h35m31.055027292s) [noise=true , address=77.90.31.10:4520 , piece_id=6SEIKD53W4W6GM6IM6MPBWG5WYFCRLPTIRWAARLK3PSHDX3IABJQ, seg_pos={0 0}, upload=463619]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:46500->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:46500->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46500->77.90.31.10:4520: use of closed network connection Dec 27 16:04:31.183950188 (528h35m31.055033223s) [noise=true , address=77.90.31.10:4520 , piece_id=6SEIKD53W4W6GM6IM6MPBWG5WYFCRLPTIRWAARLK3PSHDX3IABJQ, seg_pos={0 0}, upload=463619]: Long tail cancellation due to optimal threshold Dec 27 16:04:31.184032015 (528h35m31.05511505s) [noise=true , address=77.90.31.19:4841 , piece_id=HHKXHVITGSKYXKABTYMSFHJNRJDDHXOR3ONXIHI2KHBEMLTXAU7Q, seg_pos={0 0}, upload=463619]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:48306->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:48306->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48306->77.90.31.19:4841: use of closed network connection Dec 27 16:04:31.184036764 (528h35m31.055119799s) [noise=true , address=77.90.31.19:4841 , piece_id=HHKXHVITGSKYXKABTYMSFHJNRJDDHXOR3ONXIHI2KHBEMLTXAU7Q, seg_pos={0 0}, upload=463619]: Long tail cancellation due to optimal threshold Dec 27 16:04:31.184269146 (528h35m31.055352201s) [noise=true , address=77.90.31.11:4555 , piece_id=HE2WOM2PJXDCJBYLK43JKC2TTBFCZR4VGRCGJVEMJJN4XP2MUPLA, seg_pos={0 0}, upload=463619]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:43690->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:04:31.184293492 (528h35m31.055376537s) [noise=true , address=77.90.31.11:4555 , piece_id=HE2WOM2PJXDCJBYLK43JKC2TTBFCZR4VGRCGJVEMJJN4XP2MUPLA, seg_pos={0 0}, upload=463619]: Long tail cancellation due to optimal threshold Dec 27 16:04:31.184328940 (528h35m31.055411975s) [seg_pos={0 0}, upload=463619]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:04:31.184451153 (528h35m31.055534208s) [upload=463619]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:04:31.184458967 (528h35m31.055542012s) [upload=463619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:04:31.476265519 (528h35m31.347348574s) [upload=463619]: Done uploading segments... Dec 27 16:04:31.476290607 (528h35m31.347373652s) [upload=463619]: Upload finished. err: <nil> Duration: 30.003337066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.....XX......X................XX....................] duration: 37.691376445s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:04:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 37.691408967s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X......XX...] duration: 4.74883852s trace: https://jaeger.dev.storj.io/trace/3962867a2deb1607 FAIL (2025-12-27 16:04:34 UTC) orbiter warning: context deadline exceeded Duration: 4.748864229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X...X......X.X..X..XXX...........X.......X..X.XX..X..X....X] duration: 30.003413982s trace: https://jaeger.dev.storj.io/trace/65098e83740de5ab FAIL (2025-12-27 16:04:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: AFSE5PMCMK2WAJVSDPO6WWEGR4XIZTCAFPX4XFPXYAP4GJVQVHQQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: IPRDCJIT572XE6PNE53BZZ5QVCRCL24Z4RJSKPDMRCLLN65JUCPQ) context deadline exceeded ======LOGS====== 27 16:03:44.169548225 (528h34m44.04063127s) [upload=463608]: Starting upload Dec 27 16:03:44.169627006 (528h34m44.040710051s) [upload=463608]: Done starting upload Dec 27 16:03:44.169895216 (528h34m44.040978251s) [upload=463608]: Uploading segments... Dec 27 16:03:44.170020194 (528h34m44.041103239s) [upload=463608]: Got next segment. Inline: false Dec 27 16:03:44.170073535 (528h34m44.04115658s) [upload=463608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:03:44.170080609 (528h34m44.041163664s) [upload=463608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:03:44.416805436 (528h34m44.287888481s) [seg_pos={0 0}, upload=463608]: Begin upload segment... Dec 27 16:03:44.416882844 (528h34m44.287965879s) [seg_pos={0 0}, upload=463608]: Done begin upload segment. Dec 27 16:03:44.416938300 (528h34m44.288021345s) [noise=true , address=145.79.227.51:4411 , piece_id=SL5BRGUFSTLSXWJKFGOXVNWYCEIDM7AQ7MFHBB6O2GZDKNV3PQQA, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.416967826 (528h34m44.288050891s) [noise=true , address=145.79.227.3:4151 , piece_id=VZMHDKFHPHBWL2DW6AN7CCVQTNFNSNF2KTSHD7AY4QRTOKEAAORQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.417491012 (528h34m44.288574047s) [noise=true , address=145.79.227.17:4972 , piece_id=ZDH4Z2T4I6VGWWNJC2WMTII4MAG3F3POSK7MUZAE4KC3PG2QZEGA, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.418091786 (528h34m44.289174831s) [noise=true , address=145.79.227.35:4483 , piece_id=HE24HLGKDOEM4GTJF65TAUFW22HGK3GAI5PTVDQ4T6TMMYBFRT6Q, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.418856763 (528h34m44.289939798s) [noise=true , address=45.91.116.9:4264 , piece_id=AFSE5PMCMK2WAJVSDPO6WWEGR4XIZTCAFPX4XFPXYAP4GJVQVHQQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.418994335 (528h34m44.29007738s) [noise=true , address=145.79.227.38:4979 , piece_id=2UFA7HDZKZLKA7SU5ECT2KLKEOXZFW4JJ3UQR52S2AXPMAYNHO5A, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.419394968 (528h34m44.290478003s) [noise=true , address=145.79.227.52:4445 , piece_id=OZC3KVA7VSVWYE5HAUHV3Y7YC4XJATUW3KRBO4WVJIVRRBD2UIBQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.419864381 (528h34m44.290947416s) [noise=true , address=145.79.227.4:4185 , piece_id=LTRM6KWJ3UNYR2DCX5ZMBLKVLEO2HUS6MF6UTXFDZQXOLNZLCKZQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.420307946 (528h34m44.291390981s) [noise=true , address=77.90.31.17:4770 , piece_id=IPRDCJIT572XE6PNE53BZZ5QVCRCL24Z4RJSKPDMRCLLN65JUCPQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.420795314 (528h34m44.291878349s) [noise=true , address=145.79.227.23:4545 , piece_id=ALRMFZZQAUJI5UEQFRFHTVJOHJHTPPL4OHJBVMODPO6YGNHC2I3A, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.421219562 (528h34m44.292302597s) [noise=true , address=45.91.116.14:4190 , piece_id=DI6WRZPUPT7MTYUTN7EHGSAVMWMURYYECPSC5NBOLRDGZEDGIIYQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.421639291 (528h34m44.292722326s) [noise=true , address=145.79.227.50:4375 , piece_id=LKRAQHX4KVY5T2YPD4URJLDFSPZ66IKJGOZSLNZXRGA37VNAB7EQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.422124766 (528h34m44.293207801s) [noise=true , address=145.79.227.48:4302 , piece_id=DP3UTADC3F7F63VHWI7WOTLMJWWTUHBLGBOZY5SMF4ZW6A7L7IBQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.422299458 (528h34m44.293382503s) [noise=true , address=45.91.116.10:4082 , piece_id=CTBRU5WS6QKAJEULKA2UUEPPMQNE55IF3Z2ANNOMXOY564XZDIXA, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.422551788 (528h34m44.293634823s) [noise=true , address=145.79.227.27:4691 , piece_id=S5VGVHFJ372IOYNQMIVQDVPOVAYNMHU6UB67JAAB2TAQ4OVDHEKQ, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:44.422999431 (528h34m44.294082476s) [noise=true , address=145.79.227.39:4015 , piece_id=75A3CB3LXVJ6UIXGIYPBY3TBCM2MLDTGVTOUQPXWGUNY4B3SNNBA, seg_pos={0 0}, upload=463608]: Uploading piece... Dec 27 16:03:46.084716178 (528h34m45.955799223s) [upload=463608]: Next returned nil segment Dec 27 16:03:46.084730866 (528h34m45.955813901s) [upload=463608]: Waiting for error group managing segments... Dec 27 16:03:46.414579547 (528h34m46.285662582s) [noise=true , address=145.79.227.3:4151 , piece_id=VZMHDKFHPHBWL2DW6AN7CCVQTNFNSNF2KTSHD7AY4QRTOKEAAORQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.422596443 (528h34m46.293679498s) [noise=true , address=145.79.227.17:4972 , piece_id=ZDH4Z2T4I6VGWWNJC2WMTII4MAG3F3POSK7MUZAE4KC3PG2QZEGA, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.426451413 (528h34m46.297534448s) [noise=true , address=145.79.227.51:4411 , piece_id=SL5BRGUFSTLSXWJKFGOXVNWYCEIDM7AQ7MFHBB6O2GZDKNV3PQQA, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.450843179 (528h34m46.321926214s) [noise=true , address=145.79.227.27:4691 , piece_id=S5VGVHFJ372IOYNQMIVQDVPOVAYNMHU6UB67JAAB2TAQ4OVDHEKQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.451436509 (528h34m46.322519554s) [noise=true , address=145.79.227.50:4375 , piece_id=LKRAQHX4KVY5T2YPD4URJLDFSPZ66IKJGOZSLNZXRGA37VNAB7EQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.461228445 (528h34m46.33231149s) [noise=true , address=145.79.227.4:4185 , piece_id=LTRM6KWJ3UNYR2DCX5ZMBLKVLEO2HUS6MF6UTXFDZQXOLNZLCKZQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.464389945 (528h34m46.33547298s) [noise=true , address=145.79.227.48:4302 , piece_id=DP3UTADC3F7F63VHWI7WOTLMJWWTUHBLGBOZY5SMF4ZW6A7L7IBQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.469529351 (528h34m46.340612386s) [noise=true , address=145.79.227.23:4545 , piece_id=ALRMFZZQAUJI5UEQFRFHTVJOHJHTPPL4OHJBVMODPO6YGNHC2I3A, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.472825738 (528h34m46.343908773s) [noise=true , address=145.79.227.38:4979 , piece_id=2UFA7HDZKZLKA7SU5ECT2KLKEOXZFW4JJ3UQR52S2AXPMAYNHO5A, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.473970208 (528h34m46.345053243s) [noise=true , address=145.79.227.39:4015 , piece_id=75A3CB3LXVJ6UIXGIYPBY3TBCM2MLDTGVTOUQPXWGUNY4B3SNNBA, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.572539401 (528h34m46.443622446s) [noise=true , address=145.79.227.52:4445 , piece_id=OZC3KVA7VSVWYE5HAUHV3Y7YC4XJATUW3KRBO4WVJIVRRBD2UIBQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.659664473 (528h34m46.530747518s) [noise=true , address=77.90.31.17:4770 , piece_id=IPRDCJIT572XE6PNE53BZZ5QVCRCL24Z4RJSKPDMRCLLN65JUCPQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.694654250 (528h34m46.565737295s) [noise=true , address=45.91.116.9:4264 , piece_id=AFSE5PMCMK2WAJVSDPO6WWEGR4XIZTCAFPX4XFPXYAP4GJVQVHQQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: <nil> Dec 27 16:03:46.694677394 (528h34m46.565760439s) [seg_pos={0 0}, upload=463608]: Segment reached optimal threshold of 13 pieces. Dec 27 16:03:46.694871934 (528h34m46.565954979s) [noise=true , address=45.91.116.14:4190 , piece_id=DI6WRZPUPT7MTYUTN7EHGSAVMWMURYYECPSC5NBOLRDGZEDGIIYQ, seg_pos={0 0}, upload=463608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:47704->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:47704->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47704->45.91.116.14:4190: use of closed network connection Dec 27 16:03:46.694878857 (528h34m46.565961892s) [noise=true , address=45.91.116.14:4190 , piece_id=DI6WRZPUPT7MTYUTN7EHGSAVMWMURYYECPSC5NBOLRDGZEDGIIYQ, seg_pos={0 0}, upload=463608]: Long tail cancellation due to optimal threshold Dec 27 16:03:46.695004747 (528h34m46.566087782s) [noise=true , address=145.79.227.35:4483 , piece_id=HE24HLGKDOEM4GTJF65TAUFW22HGK3GAI5PTVDQ4T6TMMYBFRT6Q, seg_pos={0 0}, upload=463608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 27 16:03:46.695035916 (528h34m46.566118961s) [noise=true , address=145.79.227.35:4483 , piece_id=HE24HLGKDOEM4GTJF65TAUFW22HGK3GAI5PTVDQ4T6TMMYBFRT6Q, seg_pos={0 0}, upload=463608]: Long tail cancellation due to optimal threshold Dec 27 16:03:46.695057919 (528h34m46.566140954s) [noise=true , address=45.91.116.10:4082 , piece_id=CTBRU5WS6QKAJEULKA2UUEPPMQNE55IF3Z2ANNOMXOY564XZDIXA, seg_pos={0 0}, upload=463608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 16:03:46.695065483 (528h34m46.566148528s) [noise=true , address=45.91.116.10:4082 , piece_id=CTBRU5WS6QKAJEULKA2UUEPPMQNE55IF3Z2ANNOMXOY564XZDIXA, seg_pos={0 0}, upload=463608]: Long tail cancellation due to optimal threshold Dec 27 16:03:46.695085130 (528h34m46.566168165s) [seg_pos={0 0}, upload=463608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:03:46.695208275 (528h34m46.56629131s) [upload=463608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:03:46.695218784 (528h34m46.566301829s) [upload=463608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:03:46.992820211 (528h34m46.863903256s) [upload=463608]: Done uploading segments... Dec 27 16:03:46.992928006 (528h34m46.864011051s) [upload=463608]: Upload finished. err: <nil> Duration: 30.003439781s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X......XX..] duration: 5.001417349s trace: https://jaeger.dev.storj.io/trace/3962867a2deb1607 FAIL (2025-12-27 16:03:39 UTC) orbiter warning: context deadline exceeded Duration: 5.001446876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.....XX......X................XX...................] duration: 30.8867208s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:03:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 30.88694125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X...X......X.X..X..XXX...........X.......X..X.XX..X..X....] duration: 16.031502595s trace: https://jaeger.dev.storj.io/trace/4f0242888494a9f7 FAIL (2025-12-27 16:02:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA) context deadline exceeded ======LOGS====== Dec 27 15:59:11.390957921 (528h30m11.262040956s) [upload=463547]: Starting upload Dec 27 15:59:11.391043043 (528h30m11.262126078s) [upload=463547]: Done starting upload Dec 27 15:59:11.391138625 (528h30m11.26222167s) [upload=463547]: Uploading segments... Dec 27 15:59:11.391304111 (528h30m11.262387156s) [upload=463547]: Got next segment. Inline: false Dec 27 15:59:11.391344537 (528h30m11.262427583s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:59:11.391350719 (528h30m11.262433764s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:59:11.636523092 (528h30m11.507606137s) [seg_pos={0 0}, upload=463547]: Begin upload segment... Dec 27 15:59:11.636572806 (528h30m11.507655851s) [seg_pos={0 0}, upload=463547]: Done begin upload segment. Dec 27 15:59:11.636605278 (528h30m11.507688323s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.636614485 (528h30m11.50769755s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637100811 (528h30m11.508183846s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637504881 (528h30m11.508587916s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638303802 (528h30m11.509386837s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638750814 (528h30m11.509833849s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639192774 (528h30m11.510275819s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639795252 (528h30m11.510878297s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.640812820 (528h30m11.511895855s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.641320796 (528h30m11.512403831s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.642186315 (528h30m11.51326936s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643355091 (528h30m11.514438136s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643452496 (528h30m11.514535531s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.644903879 (528h30m11.515986914s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.646604197 (528h30m11.517687232s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.647567562 (528h30m11.518650597s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:13.363214221 (528h30m13.234297266s) [upload=463547]: Next returned nil segment Dec 27 15:59:13.363228188 (528h30m13.234311224s) [upload=463547]: Waiting for error group managing segments... Dec 27 15:59:13.654275475 (528h30m13.52535851s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.655574077 (528h30m13.526657112s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.656217412 (528h30m13.527300457s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.659254616 (528h30m13.530337661s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665027158 (528h30m13.536110193s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665525877 (528h30m13.536608912s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670452609 (528h30m13.541535654s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670703226 (528h30m13.541786291s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.672945035 (528h30m13.54402808s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.674408633 (528h30m13.545491668s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.677251656 (528h30m13.548334701s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.922782090 (528h30m13.793865135s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925239709 (528h30m13.796322754s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925258114 (528h30m13.796341159s) [seg_pos={0 0}, upload=463547]: Segment reached optimal threshold of 13 pieces. Dec 27 15:59:13.925466281 (528h30m13.796549316s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925475518 (528h30m13.796558553s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.925607760 (528h30m13.796690795s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925614903 (528h30m13.796697938s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926001269 (528h30m13.797084324s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.926017591 (528h30m13.797100636s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926074919 (528h30m13.797157954s) [seg_pos={0 0}, upload=463547]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:59:13.926204767 (528h30m13.797287802s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:59:13.926212191 (528h30m13.797295236s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:59:14.219791278 (528h30m14.090874323s) [upload=463547]: Done uploading segments... Dec 27 15:59:14.219809953 (528h30m14.090892998s) [upload=463547]: Upload finished. err: <nil> Duration: 16.031530788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.....XX......X................XX..................] duration: 37.861584014s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:02:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 37.861610775s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X......XX.] duration: 3.63560159s trace: https://jaeger.dev.storj.io/trace/3962867a2deb1607 FAIL (2025-12-27 16:02:30 UTC) orbiter warning: context deadline exceeded Duration: 3.636172576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X...X......X.X..X..XXX...........X.......X..X.XX..X..X...] duration: 5.673918911s trace: https://jaeger.dev.storj.io/trace/4f0242888494a9f7 FAIL (2025-12-27 16:02:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA) context deadline exceeded ======LOGS====== Dec 27 15:59:11.390957921 (528h30m11.262040956s) [upload=463547]: Starting upload Dec 27 15:59:11.391043043 (528h30m11.262126078s) [upload=463547]: Done starting upload Dec 27 15:59:11.391138625 (528h30m11.26222167s) [upload=463547]: Uploading segments... Dec 27 15:59:11.391304111 (528h30m11.262387156s) [upload=463547]: Got next segment. Inline: false Dec 27 15:59:11.391344537 (528h30m11.262427583s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:59:11.391350719 (528h30m11.262433764s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:59:11.636523092 (528h30m11.507606137s) [seg_pos={0 0}, upload=463547]: Begin upload segment... Dec 27 15:59:11.636572806 (528h30m11.507655851s) [seg_pos={0 0}, upload=463547]: Done begin upload segment. Dec 27 15:59:11.636605278 (528h30m11.507688323s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.636614485 (528h30m11.50769755s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637100811 (528h30m11.508183846s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637504881 (528h30m11.508587916s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638303802 (528h30m11.509386837s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638750814 (528h30m11.509833849s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639192774 (528h30m11.510275819s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639795252 (528h30m11.510878297s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.640812820 (528h30m11.511895855s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.641320796 (528h30m11.512403831s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.642186315 (528h30m11.51326936s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643355091 (528h30m11.514438136s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643452496 (528h30m11.514535531s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.644903879 (528h30m11.515986914s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.646604197 (528h30m11.517687232s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.647567562 (528h30m11.518650597s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:13.363214221 (528h30m13.234297266s) [upload=463547]: Next returned nil segment Dec 27 15:59:13.363228188 (528h30m13.234311224s) [upload=463547]: Waiting for error group managing segments... Dec 27 15:59:13.654275475 (528h30m13.52535851s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.655574077 (528h30m13.526657112s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.656217412 (528h30m13.527300457s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.659254616 (528h30m13.530337661s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665027158 (528h30m13.536110193s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665525877 (528h30m13.536608912s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670452609 (528h30m13.541535654s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670703226 (528h30m13.541786291s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.672945035 (528h30m13.54402808s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.674408633 (528h30m13.545491668s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.677251656 (528h30m13.548334701s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.922782090 (528h30m13.793865135s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925239709 (528h30m13.796322754s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925258114 (528h30m13.796341159s) [seg_pos={0 0}, upload=463547]: Segment reached optimal threshold of 13 pieces. Dec 27 15:59:13.925466281 (528h30m13.796549316s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925475518 (528h30m13.796558553s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.925607760 (528h30m13.796690795s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925614903 (528h30m13.796697938s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926001269 (528h30m13.797084324s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.926017591 (528h30m13.797100636s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926074919 (528h30m13.797157954s) [seg_pos={0 0}, upload=463547]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:59:13.926204767 (528h30m13.797287802s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:59:13.926212191 (528h30m13.797295236s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:59:14.219791278 (528h30m14.090874323s) [upload=463547]: Done uploading segments... Dec 27 15:59:14.219809953 (528h30m14.090892998s) [upload=463547]: Upload finished. err: <nil> Duration: 5.673959558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.....XX......X................XX.................] duration: 42.573741183s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:01:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 42.573768715s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X......XX] duration: 5.000830381s trace: https://jaeger.dev.storj.io/trace/3962867a2deb1607 FAIL (2025-12-27 16:01:20 UTC) orbiter warning: context deadline exceeded Duration: 5.000858576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X...X......X.X..X..XXX...........X.......X..X.XX..X..X..] duration: 5.92435781s trace: https://jaeger.dev.storj.io/trace/4f0242888494a9f7 FAIL (2025-12-27 16:00:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA) context deadline exceeded ======LOGS====== Dec 27 15:59:11.390957921 (528h30m11.262040956s) [upload=463547]: Starting upload Dec 27 15:59:11.391043043 (528h30m11.262126078s) [upload=463547]: Done starting upload Dec 27 15:59:11.391138625 (528h30m11.26222167s) [upload=463547]: Uploading segments... Dec 27 15:59:11.391304111 (528h30m11.262387156s) [upload=463547]: Got next segment. Inline: false Dec 27 15:59:11.391344537 (528h30m11.262427583s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:59:11.391350719 (528h30m11.262433764s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:59:11.636523092 (528h30m11.507606137s) [seg_pos={0 0}, upload=463547]: Begin upload segment... Dec 27 15:59:11.636572806 (528h30m11.507655851s) [seg_pos={0 0}, upload=463547]: Done begin upload segment. Dec 27 15:59:11.636605278 (528h30m11.507688323s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.636614485 (528h30m11.50769755s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637100811 (528h30m11.508183846s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637504881 (528h30m11.508587916s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638303802 (528h30m11.509386837s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638750814 (528h30m11.509833849s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639192774 (528h30m11.510275819s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639795252 (528h30m11.510878297s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.640812820 (528h30m11.511895855s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.641320796 (528h30m11.512403831s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.642186315 (528h30m11.51326936s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643355091 (528h30m11.514438136s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643452496 (528h30m11.514535531s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.644903879 (528h30m11.515986914s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.646604197 (528h30m11.517687232s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.647567562 (528h30m11.518650597s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:13.363214221 (528h30m13.234297266s) [upload=463547]: Next returned nil segment Dec 27 15:59:13.363228188 (528h30m13.234311224s) [upload=463547]: Waiting for error group managing segments... Dec 27 15:59:13.654275475 (528h30m13.52535851s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.655574077 (528h30m13.526657112s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.656217412 (528h30m13.527300457s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.659254616 (528h30m13.530337661s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665027158 (528h30m13.536110193s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665525877 (528h30m13.536608912s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670452609 (528h30m13.541535654s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670703226 (528h30m13.541786291s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.672945035 (528h30m13.54402808s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.674408633 (528h30m13.545491668s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.677251656 (528h30m13.548334701s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.922782090 (528h30m13.793865135s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925239709 (528h30m13.796322754s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925258114 (528h30m13.796341159s) [seg_pos={0 0}, upload=463547]: Segment reached optimal threshold of 13 pieces. Dec 27 15:59:13.925466281 (528h30m13.796549316s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925475518 (528h30m13.796558553s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.925607760 (528h30m13.796690795s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925614903 (528h30m13.796697938s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926001269 (528h30m13.797084324s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.926017591 (528h30m13.797100636s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926074919 (528h30m13.797157954s) [seg_pos={0 0}, upload=463547]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:59:13.926204767 (528h30m13.797287802s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:59:13.926212191 (528h30m13.797295236s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:59:14.219791278 (528h30m14.090874323s) [upload=463547]: Done uploading segments... Dec 27 15:59:14.219809953 (528h30m14.090892998s) [upload=463547]: Upload finished. err: <nil> Duration: 5.924391764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X.....XX......X................XX................] duration: 34.768966891s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:00:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 34.768995508s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X......X] duration: 5.002720957s trace: https://jaeger.dev.storj.io/trace/7f463538f1c2cb4c FAIL (2025-12-27 16:00:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002749182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X...X......X.X..X..XXX...........X.......X..X.XX..X..X.] duration: 5.714722656s trace: https://jaeger.dev.storj.io/trace/4f0242888494a9f7 FAIL (2025-12-27 15:59:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA) context deadline exceeded ======LOGS====== Dec 27 15:59:11.390957921 (528h30m11.262040956s) [upload=463547]: Starting upload Dec 27 15:59:11.391043043 (528h30m11.262126078s) [upload=463547]: Done starting upload Dec 27 15:59:11.391138625 (528h30m11.26222167s) [upload=463547]: Uploading segments... Dec 27 15:59:11.391304111 (528h30m11.262387156s) [upload=463547]: Got next segment. Inline: false Dec 27 15:59:11.391344537 (528h30m11.262427583s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:59:11.391350719 (528h30m11.262433764s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:59:11.636523092 (528h30m11.507606137s) [seg_pos={0 0}, upload=463547]: Begin upload segment... Dec 27 15:59:11.636572806 (528h30m11.507655851s) [seg_pos={0 0}, upload=463547]: Done begin upload segment. Dec 27 15:59:11.636605278 (528h30m11.507688323s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.636614485 (528h30m11.50769755s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637100811 (528h30m11.508183846s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637504881 (528h30m11.508587916s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638303802 (528h30m11.509386837s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638750814 (528h30m11.509833849s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639192774 (528h30m11.510275819s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639795252 (528h30m11.510878297s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.640812820 (528h30m11.511895855s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.641320796 (528h30m11.512403831s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.642186315 (528h30m11.51326936s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643355091 (528h30m11.514438136s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643452496 (528h30m11.514535531s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.644903879 (528h30m11.515986914s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.646604197 (528h30m11.517687232s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.647567562 (528h30m11.518650597s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:13.363214221 (528h30m13.234297266s) [upload=463547]: Next returned nil segment Dec 27 15:59:13.363228188 (528h30m13.234311224s) [upload=463547]: Waiting for error group managing segments... Dec 27 15:59:13.654275475 (528h30m13.52535851s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.655574077 (528h30m13.526657112s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.656217412 (528h30m13.527300457s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.659254616 (528h30m13.530337661s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665027158 (528h30m13.536110193s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665525877 (528h30m13.536608912s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670452609 (528h30m13.541535654s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670703226 (528h30m13.541786291s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.672945035 (528h30m13.54402808s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.674408633 (528h30m13.545491668s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.677251656 (528h30m13.548334701s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.922782090 (528h30m13.793865135s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925239709 (528h30m13.796322754s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925258114 (528h30m13.796341159s) [seg_pos={0 0}, upload=463547]: Segment reached optimal threshold of 13 pieces. Dec 27 15:59:13.925466281 (528h30m13.796549316s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925475518 (528h30m13.796558553s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.925607760 (528h30m13.796690795s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925614903 (528h30m13.796697938s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926001269 (528h30m13.797084324s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.926017591 (528h30m13.797100636s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926074919 (528h30m13.797157954s) [seg_pos={0 0}, upload=463547]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:59:13.926204767 (528h30m13.797287802s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:59:13.926212191 (528h30m13.797295236s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:59:14.219791278 (528h30m14.090874323s) [upload=463547]: Done uploading segments... Dec 27 15:59:14.219809953 (528h30m14.090892998s) [upload=463547]: Upload finished. err: <nil> Duration: 5.714736763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.....X...X......X.X..X..XXX...........X.......X..X.XX..X..X] duration: 30.017274712s trace: https://jaeger.dev.storj.io/trace/4f0242888494a9f7 FAIL (2025-12-27 15:59:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA) context deadline exceeded ======LOGS====== Dec 27 15:59:11.390957921 (528h30m11.262040956s) [upload=463547]: Starting upload Dec 27 15:59:11.391043043 (528h30m11.262126078s) [upload=463547]: Done starting upload Dec 27 15:59:11.391138625 (528h30m11.26222167s) [upload=463547]: Uploading segments... Dec 27 15:59:11.391304111 (528h30m11.262387156s) [upload=463547]: Got next segment. Inline: false Dec 27 15:59:11.391344537 (528h30m11.262427583s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:59:11.391350719 (528h30m11.262433764s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:59:11.636523092 (528h30m11.507606137s) [seg_pos={0 0}, upload=463547]: Begin upload segment... Dec 27 15:59:11.636572806 (528h30m11.507655851s) [seg_pos={0 0}, upload=463547]: Done begin upload segment. Dec 27 15:59:11.636605278 (528h30m11.507688323s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.636614485 (528h30m11.50769755s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637100811 (528h30m11.508183846s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.637504881 (528h30m11.508587916s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638303802 (528h30m11.509386837s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.638750814 (528h30m11.509833849s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639192774 (528h30m11.510275819s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.639795252 (528h30m11.510878297s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.640812820 (528h30m11.511895855s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.641320796 (528h30m11.512403831s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.642186315 (528h30m11.51326936s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643355091 (528h30m11.514438136s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.643452496 (528h30m11.514535531s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.644903879 (528h30m11.515986914s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.646604197 (528h30m11.517687232s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:11.647567562 (528h30m11.518650597s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Uploading piece... Dec 27 15:59:13.363214221 (528h30m13.234297266s) [upload=463547]: Next returned nil segment Dec 27 15:59:13.363228188 (528h30m13.234311224s) [upload=463547]: Waiting for error group managing segments... Dec 27 15:59:13.654275475 (528h30m13.52535851s) [noise=true , address=145.79.227.32:4834 , piece_id=VMCPVZYXAJBOAJXIDS4ZSTNZHBOEUVCB57KPHQCXOHQLR25LMVYA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.655574077 (528h30m13.526657112s) [noise=true , address=145.79.227.52:4446 , piece_id=G7CAQUF2AYJLPX6P36GVN7FAFSEEXQ6K6TW6WFE6YGEOEITFUF6Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.656217412 (528h30m13.527300457s) [noise=true , address=145.79.227.16:4912 , piece_id=MLP5Q65JM3YJAXJZIWPOPB6JJYIFDXK2ZA272UDFR6PTZVJ7JUGQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.659254616 (528h30m13.530337661s) [noise=true , address=145.79.227.29:4942 , piece_id=4EK6NXHN4JUEA2UZ2TQ62JBXD22AHKLYK3TAHVWJJ5UYXJZ42BRQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665027158 (528h30m13.536110193s) [noise=true , address=145.79.227.3:4149 , piece_id=Z57JLHNUFHYGMWBHOETF7IJUGBEVI2LOW7TSCKGIK6CTGW2FYRKQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.665525877 (528h30m13.536608912s) [noise=true , address=145.79.227.17:4971 , piece_id=XXH7FBJ2I7KBU33FHWM24YY373655RNRUENEZHTJQQ2AU6TA7X7Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670452609 (528h30m13.541535654s) [noise=true , address=145.79.227.20:4438 , piece_id=XGITOJPPS52TA5BCLOWHT76C46UUIWNFER35A3WF76SWT3CRTHSQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.670703226 (528h30m13.541786291s) [noise=true , address=145.79.227.2:4114 , piece_id=2EVYS4FC2PESKMC6GISXI2XEAJWT33PLT4X44ERJG6WSPNSJ2RTA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.672945035 (528h30m13.54402808s) [noise=true , address=145.79.227.24:4581 , piece_id=ZSN5Q2CLWI7GMF733NZPH2E2D3TTUGH46ZPG6MG4G4WF7E4RIANA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.674408633 (528h30m13.545491668s) [noise=true , address=145.79.227.13:4802 , piece_id=MP5LMHI3XCESI54MXGST2R3MDREPLQ7XC5GDZ6HVPLWAYUPJ4ZVQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.677251656 (528h30m13.548334701s) [noise=true , address=145.79.227.41:4086 , piece_id=RLHBF4GIXWOHUES6FOVA5ZYHOL32XW2DRFUTVV3VNEQ2DVIGIB4Q, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.922782090 (528h30m13.793865135s) [noise=true , address=77.90.31.19:4841 , piece_id=2DBF4VZQY7WUCKQ7IX3OUARDVS632O3UYCUSEJG5BSUGMXD4MTUA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925239709 (528h30m13.796322754s) [noise=true , address=45.91.116.5:4297 , piece_id=MWVHUCFDQJIVV5323TGLVKYZIH7YV7BKECDJ7IFWCAMIKWBS6QIA, seg_pos={0 0}, upload=463547]: Done uploading piece. err: <nil> Dec 27 15:59:13.925258114 (528h30m13.796341159s) [seg_pos={0 0}, upload=463547]: Segment reached optimal threshold of 13 pieces. Dec 27 15:59:13.925466281 (528h30m13.796549316s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925475518 (528h30m13.796558553s) [noise=true , address=45.91.116.12:4228 , piece_id=PWVE2GDLQIDPF47C3HIYOMACAABZM5YZTTWBQES7COSDOKRDVITQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.925607760 (528h30m13.796690795s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.925614903 (528h30m13.796697938s) [noise=true , address=45.91.116.10:4083 , piece_id=QQ5CZYPTCIMANXJYUBO26M2N4ZWO4GWC35G65U2UDWLWMKPMHUOQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926001269 (528h30m13.797084324s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:59:13.926017591 (528h30m13.797100636s) [noise=true , address=145.79.227.30:4761 , piece_id=TC2OV6ZR7QJM4VRDAD2XUSNHHSEX5PCQAUG5ZLBVKEOTOGGEKCPQ, seg_pos={0 0}, upload=463547]: Long tail cancellation due to optimal threshold Dec 27 15:59:13.926074919 (528h30m13.797157954s) [seg_pos={0 0}, upload=463547]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:59:13.926204767 (528h30m13.797287802s) [upload=463547]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:59:13.926212191 (528h30m13.797295236s) [upload=463547]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:59:14.219791278 (528h30m14.090874323s) [upload=463547]: Done uploading segments... Dec 27 15:59:14.219809953 (528h30m14.090892998s) [upload=463547]: Upload finished. err: <nil> Duration: 30.017303126s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X......] duration: 3.711629188s trace: https://jaeger.dev.storj.io/trace/365412d41f860e21 FAIL (2025-12-27 15:59:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.711649848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X.....XX......X................XX...............] duration: 31.00304346s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:59:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 31.003059861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.....X...X......X.X..X..XXX...........X.......X..X.XX..X..] duration: 5.942453747s trace: https://jaeger.dev.storj.io/trace/4d43a1d6afd8785c FAIL (2025-12-27 15:58:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA) context deadline exceeded ======LOGS====== c 27 15:56:37.197309231 (528h27m37.068392276s) [upload=463507]: Starting upload Dec 27 15:56:37.197389514 (528h27m37.068472559s) [upload=463507]: Done starting upload Dec 27 15:56:37.197561050 (528h27m37.068644095s) [upload=463507]: Uploading segments... Dec 27 15:56:37.197792461 (528h27m37.068875496s) [upload=463507]: Got next segment. Inline: false Dec 27 15:56:37.197865110 (528h27m37.068948155s) [upload=463507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:56:37.197872103 (528h27m37.068955148s) [upload=463507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:56:37.439395246 (528h27m37.310478291s) [seg_pos={0 0}, upload=463507]: Begin upload segment... Dec 27 15:56:37.439464508 (528h27m37.310547543s) [seg_pos={0 0}, upload=463507]: Done begin upload segment. Dec 27 15:56:37.439524142 (528h27m37.310607177s) [noise=true , address=145.79.227.45:4230 , piece_id=A4YDOAANYATP457CNGDRH5T25ULGKHTKUPTHFSZBZEINMGWECPIA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.439594927 (528h27m37.310677962s) [noise=true , address=145.79.227.52:4445 , piece_id=XAR7FXE52GRZND4LAD5QQ76UCAA3R2WWPXWF3U77K6S4QQH5CU4A, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.440184469 (528h27m37.311267515s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.440597646 (528h27m37.311680681s) [noise=true , address=145.79.227.13:4804 , piece_id=TKOD47CAYP66ZMQHVZH7BJCZ2D3HT7QQI3MUMNAXOBK7AAGP3IIQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.441024960 (528h27m37.312107995s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.441551061 (528h27m37.312634106s) [noise=true , address=145.79.227.15:4874 , piece_id=LCOSGDSBU5XA4Y24ITKZ2YEJ7Q62UOUKXIZHPH2AZGTQSDJNV5TA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.442240785 (528h27m37.31332382s) [noise=true , address=45.91.116.6:4974 , piece_id=O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.443917007 (528h27m37.315000042s) [noise=true , address=77.90.31.11:4554 , piece_id=7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.445497666 (528h27m37.316580701s) [noise=true , address=145.79.227.51:4410 , piece_id=NFD7F2PM7SFEDHJ27TRDQYONANEHRQ6FTWRQABNLM3IAAHFYAGVQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.447266815 (528h27m37.31834986s) [noise=true , address=145.79.227.10:4403 , piece_id=VGL3NJABPGYHCG7WY56YEBFHI5VJFSQEKVRT23KRW6B327KLYKQA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.448786599 (528h27m37.319869644s) [noise=true , address=145.79.227.16:4911 , piece_id=TQ4CXBMRVRP7JB4GZKWOXDCQIQVMZJXUZ346MBUPUESCCQJY53EQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.449228802 (528h27m37.320311837s) [noise=true , address=45.91.116.10:4081 , piece_id=AEO2DIWH44M4POUNXVJXT7PQG5QI4IOFAQRWROTV5ICZTJW72NDA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.450877641 (528h27m37.321960676s) [noise=true , address=145.79.227.3:4150 , piece_id=QK4YPFONOJ5LYLT76JO3LXBFHEMMTZ65VR66QUX4LFG4NC3DQ7HQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.452286183 (528h27m37.323369228s) [noise=true , address=145.79.227.17:4970 , piece_id=YN73TI6N63IBCVV5PGJA23PNPC4ZVXWAO65GRJZZP4OK45MFO7LQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.452805722 (528h27m37.323888767s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.453111044 (528h27m37.324194079s) [noise=true , address=145.79.227.18:4007 , piece_id=RWNT2X7NGRDJQ7O34YNQNJVQVXES7UZMSWGAX7LUXVXIWMRYIBXA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:39.147141372 (528h27m39.018224407s) [upload=463507]: Next returned nil segment Dec 27 15:56:39.147158665 (528h27m39.01824171s) [upload=463507]: Waiting for error group managing segments... Dec 27 15:56:39.435732281 (528h27m39.306815326s) [noise=true , address=145.79.227.45:4230 , piece_id=A4YDOAANYATP457CNGDRH5T25ULGKHTKUPTHFSZBZEINMGWECPIA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.439969059 (528h27m39.311052094s) [noise=true , address=145.79.227.51:4410 , piece_id=NFD7F2PM7SFEDHJ27TRDQYONANEHRQ6FTWRQABNLM3IAAHFYAGVQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.440872740 (528h27m39.311955785s) [noise=true , address=145.79.227.10:4403 , piece_id=VGL3NJABPGYHCG7WY56YEBFHI5VJFSQEKVRT23KRW6B327KLYKQA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.455024286 (528h27m39.326107321s) [noise=true , address=145.79.227.16:4911 , piece_id=TQ4CXBMRVRP7JB4GZKWOXDCQIQVMZJXUZ346MBUPUESCCQJY53EQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.455787740 (528h27m39.326870785s) [noise=true , address=145.79.227.3:4150 , piece_id=QK4YPFONOJ5LYLT76JO3LXBFHEMMTZ65VR66QUX4LFG4NC3DQ7HQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.456260731 (528h27m39.327343776s) [noise=true , address=145.79.227.15:4874 , piece_id=LCOSGDSBU5XA4Y24ITKZ2YEJ7Q62UOUKXIZHPH2AZGTQSDJNV5TA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.461398053 (528h27m39.332481098s) [noise=true , address=145.79.227.52:4445 , piece_id=XAR7FXE52GRZND4LAD5QQ76UCAA3R2WWPXWF3U77K6S4QQH5CU4A, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.464695132 (528h27m39.335778167s) [noise=true , address=145.79.227.17:4970 , piece_id=YN73TI6N63IBCVV5PGJA23PNPC4ZVXWAO65GRJZZP4OK45MFO7LQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.474375355 (528h27m39.34545839s) [noise=true , address=145.79.227.13:4804 , piece_id=TKOD47CAYP66ZMQHVZH7BJCZ2D3HT7QQI3MUMNAXOBK7AAGP3IIQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.484112898 (528h27m39.355195933s) [noise=true , address=145.79.227.18:4007 , piece_id=RWNT2X7NGRDJQ7O34YNQNJVQVXES7UZMSWGAX7LUXVXIWMRYIBXA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.618260909 (528h27m39.489343954s) [noise=true , address=77.90.31.11:4554 , piece_id=7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.711842235 (528h27m39.58292528s) [noise=true , address=45.91.116.10:4081 , piece_id=AEO2DIWH44M4POUNXVJXT7PQG5QI4IOFAQRWROTV5ICZTJW72NDA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.795897832 (528h27m39.666980877s) [noise=true , address=45.91.116.6:4974 , piece_id=O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.795923310 (528h27m39.667006355s) [seg_pos={0 0}, upload=463507]: Segment reached optimal threshold of 13 pieces. Dec 27 15:56:39.796163628 (528h27m39.667246673s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 15:56:39.796172264 (528h27m39.667255299s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796350234 (528h27m39.667433279s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 15:56:39.796366124 (528h27m39.667449169s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796487524 (528h27m39.667570569s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection Dec 27 15:56:39.796497454 (528h27m39.667580499s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796518253 (528h27m39.667601298s) [seg_pos={0 0}, upload=463507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:56:39.796617292 (528h27m39.667700337s) [upload=463507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:56:39.796624415 (528h27m39.66770745s) [upload=463507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:56:40.092401729 (528h27m39.963484774s) [upload=463507]: Done uploading segments... Dec 27 15:56:40.092421216 (528h27m39.963504261s) [upload=463507]: Upload finished. err: <nil> Duration: 5.942480117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X.....XX......X................XX..............] duration: 42.449163081s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:58:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 42.449176326s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X.....] duration: 3.65087333s trace: https://jaeger.dev.storj.io/trace/365412d41f860e21 FAIL (2025-12-27 15:58:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.650948251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X.....XX......X................XX.............] duration: 53.243021748s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:58:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 53.243036796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.....X...X......X.X..X..XXX...........X.......X..X.XX..X.] duration: 5.62074112s trace: https://jaeger.dev.storj.io/trace/4d43a1d6afd8785c FAIL (2025-12-27 15:57:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA) context deadline exceeded ======LOGS====== c 27 15:56:37.197309231 (528h27m37.068392276s) [upload=463507]: Starting upload Dec 27 15:56:37.197389514 (528h27m37.068472559s) [upload=463507]: Done starting upload Dec 27 15:56:37.197561050 (528h27m37.068644095s) [upload=463507]: Uploading segments... Dec 27 15:56:37.197792461 (528h27m37.068875496s) [upload=463507]: Got next segment. Inline: false Dec 27 15:56:37.197865110 (528h27m37.068948155s) [upload=463507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:56:37.197872103 (528h27m37.068955148s) [upload=463507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:56:37.439395246 (528h27m37.310478291s) [seg_pos={0 0}, upload=463507]: Begin upload segment... Dec 27 15:56:37.439464508 (528h27m37.310547543s) [seg_pos={0 0}, upload=463507]: Done begin upload segment. Dec 27 15:56:37.439524142 (528h27m37.310607177s) [noise=true , address=145.79.227.45:4230 , piece_id=A4YDOAANYATP457CNGDRH5T25ULGKHTKUPTHFSZBZEINMGWECPIA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.439594927 (528h27m37.310677962s) [noise=true , address=145.79.227.52:4445 , piece_id=XAR7FXE52GRZND4LAD5QQ76UCAA3R2WWPXWF3U77K6S4QQH5CU4A, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.440184469 (528h27m37.311267515s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.440597646 (528h27m37.311680681s) [noise=true , address=145.79.227.13:4804 , piece_id=TKOD47CAYP66ZMQHVZH7BJCZ2D3HT7QQI3MUMNAXOBK7AAGP3IIQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.441024960 (528h27m37.312107995s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.441551061 (528h27m37.312634106s) [noise=true , address=145.79.227.15:4874 , piece_id=LCOSGDSBU5XA4Y24ITKZ2YEJ7Q62UOUKXIZHPH2AZGTQSDJNV5TA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.442240785 (528h27m37.31332382s) [noise=true , address=45.91.116.6:4974 , piece_id=O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.443917007 (528h27m37.315000042s) [noise=true , address=77.90.31.11:4554 , piece_id=7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.445497666 (528h27m37.316580701s) [noise=true , address=145.79.227.51:4410 , piece_id=NFD7F2PM7SFEDHJ27TRDQYONANEHRQ6FTWRQABNLM3IAAHFYAGVQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.447266815 (528h27m37.31834986s) [noise=true , address=145.79.227.10:4403 , piece_id=VGL3NJABPGYHCG7WY56YEBFHI5VJFSQEKVRT23KRW6B327KLYKQA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.448786599 (528h27m37.319869644s) [noise=true , address=145.79.227.16:4911 , piece_id=TQ4CXBMRVRP7JB4GZKWOXDCQIQVMZJXUZ346MBUPUESCCQJY53EQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.449228802 (528h27m37.320311837s) [noise=true , address=45.91.116.10:4081 , piece_id=AEO2DIWH44M4POUNXVJXT7PQG5QI4IOFAQRWROTV5ICZTJW72NDA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.450877641 (528h27m37.321960676s) [noise=true , address=145.79.227.3:4150 , piece_id=QK4YPFONOJ5LYLT76JO3LXBFHEMMTZ65VR66QUX4LFG4NC3DQ7HQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.452286183 (528h27m37.323369228s) [noise=true , address=145.79.227.17:4970 , piece_id=YN73TI6N63IBCVV5PGJA23PNPC4ZVXWAO65GRJZZP4OK45MFO7LQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.452805722 (528h27m37.323888767s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.453111044 (528h27m37.324194079s) [noise=true , address=145.79.227.18:4007 , piece_id=RWNT2X7NGRDJQ7O34YNQNJVQVXES7UZMSWGAX7LUXVXIWMRYIBXA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:39.147141372 (528h27m39.018224407s) [upload=463507]: Next returned nil segment Dec 27 15:56:39.147158665 (528h27m39.01824171s) [upload=463507]: Waiting for error group managing segments... Dec 27 15:56:39.435732281 (528h27m39.306815326s) [noise=true , address=145.79.227.45:4230 , piece_id=A4YDOAANYATP457CNGDRH5T25ULGKHTKUPTHFSZBZEINMGWECPIA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.439969059 (528h27m39.311052094s) [noise=true , address=145.79.227.51:4410 , piece_id=NFD7F2PM7SFEDHJ27TRDQYONANEHRQ6FTWRQABNLM3IAAHFYAGVQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.440872740 (528h27m39.311955785s) [noise=true , address=145.79.227.10:4403 , piece_id=VGL3NJABPGYHCG7WY56YEBFHI5VJFSQEKVRT23KRW6B327KLYKQA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.455024286 (528h27m39.326107321s) [noise=true , address=145.79.227.16:4911 , piece_id=TQ4CXBMRVRP7JB4GZKWOXDCQIQVMZJXUZ346MBUPUESCCQJY53EQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.455787740 (528h27m39.326870785s) [noise=true , address=145.79.227.3:4150 , piece_id=QK4YPFONOJ5LYLT76JO3LXBFHEMMTZ65VR66QUX4LFG4NC3DQ7HQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.456260731 (528h27m39.327343776s) [noise=true , address=145.79.227.15:4874 , piece_id=LCOSGDSBU5XA4Y24ITKZ2YEJ7Q62UOUKXIZHPH2AZGTQSDJNV5TA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.461398053 (528h27m39.332481098s) [noise=true , address=145.79.227.52:4445 , piece_id=XAR7FXE52GRZND4LAD5QQ76UCAA3R2WWPXWF3U77K6S4QQH5CU4A, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.464695132 (528h27m39.335778167s) [noise=true , address=145.79.227.17:4970 , piece_id=YN73TI6N63IBCVV5PGJA23PNPC4ZVXWAO65GRJZZP4OK45MFO7LQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.474375355 (528h27m39.34545839s) [noise=true , address=145.79.227.13:4804 , piece_id=TKOD47CAYP66ZMQHVZH7BJCZ2D3HT7QQI3MUMNAXOBK7AAGP3IIQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.484112898 (528h27m39.355195933s) [noise=true , address=145.79.227.18:4007 , piece_id=RWNT2X7NGRDJQ7O34YNQNJVQVXES7UZMSWGAX7LUXVXIWMRYIBXA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.618260909 (528h27m39.489343954s) [noise=true , address=77.90.31.11:4554 , piece_id=7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.711842235 (528h27m39.58292528s) [noise=true , address=45.91.116.10:4081 , piece_id=AEO2DIWH44M4POUNXVJXT7PQG5QI4IOFAQRWROTV5ICZTJW72NDA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.795897832 (528h27m39.666980877s) [noise=true , address=45.91.116.6:4974 , piece_id=O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.795923310 (528h27m39.667006355s) [seg_pos={0 0}, upload=463507]: Segment reached optimal threshold of 13 pieces. Dec 27 15:56:39.796163628 (528h27m39.667246673s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 15:56:39.796172264 (528h27m39.667255299s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796350234 (528h27m39.667433279s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 15:56:39.796366124 (528h27m39.667449169s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796487524 (528h27m39.667570569s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection Dec 27 15:56:39.796497454 (528h27m39.667580499s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796518253 (528h27m39.667601298s) [seg_pos={0 0}, upload=463507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:56:39.796617292 (528h27m39.667700337s) [upload=463507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:56:39.796624415 (528h27m39.66770745s) [upload=463507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:56:40.092401729 (528h27m39.963484774s) [upload=463507]: Done uploading segments... Dec 27 15:56:40.092421216 (528h27m39.963504261s) [upload=463507]: Upload finished. err: <nil> Duration: 5.620768011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X....] duration: 5.001301754s trace: https://jaeger.dev.storj.io/trace/365412d41f860e21 FAIL (2025-12-27 15:57:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001408126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X.....XX......X................XX............] duration: 46.414949139s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:57:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 46.414978384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X.....X...X......X.X..X..XXX...........X.......X..X.XX..X] duration: 30.003163832s trace: https://jaeger.dev.storj.io/trace/4d43a1d6afd8785c FAIL (2025-12-27 15:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA) context deadline exceeded ======LOGS====== c 27 15:56:37.197309231 (528h27m37.068392276s) [upload=463507]: Starting upload Dec 27 15:56:37.197389514 (528h27m37.068472559s) [upload=463507]: Done starting upload Dec 27 15:56:37.197561050 (528h27m37.068644095s) [upload=463507]: Uploading segments... Dec 27 15:56:37.197792461 (528h27m37.068875496s) [upload=463507]: Got next segment. Inline: false Dec 27 15:56:37.197865110 (528h27m37.068948155s) [upload=463507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:56:37.197872103 (528h27m37.068955148s) [upload=463507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:56:37.439395246 (528h27m37.310478291s) [seg_pos={0 0}, upload=463507]: Begin upload segment... Dec 27 15:56:37.439464508 (528h27m37.310547543s) [seg_pos={0 0}, upload=463507]: Done begin upload segment. Dec 27 15:56:37.439524142 (528h27m37.310607177s) [noise=true , address=145.79.227.45:4230 , piece_id=A4YDOAANYATP457CNGDRH5T25ULGKHTKUPTHFSZBZEINMGWECPIA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.439594927 (528h27m37.310677962s) [noise=true , address=145.79.227.52:4445 , piece_id=XAR7FXE52GRZND4LAD5QQ76UCAA3R2WWPXWF3U77K6S4QQH5CU4A, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.440184469 (528h27m37.311267515s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.440597646 (528h27m37.311680681s) [noise=true , address=145.79.227.13:4804 , piece_id=TKOD47CAYP66ZMQHVZH7BJCZ2D3HT7QQI3MUMNAXOBK7AAGP3IIQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.441024960 (528h27m37.312107995s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.441551061 (528h27m37.312634106s) [noise=true , address=145.79.227.15:4874 , piece_id=LCOSGDSBU5XA4Y24ITKZ2YEJ7Q62UOUKXIZHPH2AZGTQSDJNV5TA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.442240785 (528h27m37.31332382s) [noise=true , address=45.91.116.6:4974 , piece_id=O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.443917007 (528h27m37.315000042s) [noise=true , address=77.90.31.11:4554 , piece_id=7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.445497666 (528h27m37.316580701s) [noise=true , address=145.79.227.51:4410 , piece_id=NFD7F2PM7SFEDHJ27TRDQYONANEHRQ6FTWRQABNLM3IAAHFYAGVQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.447266815 (528h27m37.31834986s) [noise=true , address=145.79.227.10:4403 , piece_id=VGL3NJABPGYHCG7WY56YEBFHI5VJFSQEKVRT23KRW6B327KLYKQA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.448786599 (528h27m37.319869644s) [noise=true , address=145.79.227.16:4911 , piece_id=TQ4CXBMRVRP7JB4GZKWOXDCQIQVMZJXUZ346MBUPUESCCQJY53EQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.449228802 (528h27m37.320311837s) [noise=true , address=45.91.116.10:4081 , piece_id=AEO2DIWH44M4POUNXVJXT7PQG5QI4IOFAQRWROTV5ICZTJW72NDA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.450877641 (528h27m37.321960676s) [noise=true , address=145.79.227.3:4150 , piece_id=QK4YPFONOJ5LYLT76JO3LXBFHEMMTZ65VR66QUX4LFG4NC3DQ7HQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.452286183 (528h27m37.323369228s) [noise=true , address=145.79.227.17:4970 , piece_id=YN73TI6N63IBCVV5PGJA23PNPC4ZVXWAO65GRJZZP4OK45MFO7LQ, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.452805722 (528h27m37.323888767s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:37.453111044 (528h27m37.324194079s) [noise=true , address=145.79.227.18:4007 , piece_id=RWNT2X7NGRDJQ7O34YNQNJVQVXES7UZMSWGAX7LUXVXIWMRYIBXA, seg_pos={0 0}, upload=463507]: Uploading piece... Dec 27 15:56:39.147141372 (528h27m39.018224407s) [upload=463507]: Next returned nil segment Dec 27 15:56:39.147158665 (528h27m39.01824171s) [upload=463507]: Waiting for error group managing segments... Dec 27 15:56:39.435732281 (528h27m39.306815326s) [noise=true , address=145.79.227.45:4230 , piece_id=A4YDOAANYATP457CNGDRH5T25ULGKHTKUPTHFSZBZEINMGWECPIA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.439969059 (528h27m39.311052094s) [noise=true , address=145.79.227.51:4410 , piece_id=NFD7F2PM7SFEDHJ27TRDQYONANEHRQ6FTWRQABNLM3IAAHFYAGVQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.440872740 (528h27m39.311955785s) [noise=true , address=145.79.227.10:4403 , piece_id=VGL3NJABPGYHCG7WY56YEBFHI5VJFSQEKVRT23KRW6B327KLYKQA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.455024286 (528h27m39.326107321s) [noise=true , address=145.79.227.16:4911 , piece_id=TQ4CXBMRVRP7JB4GZKWOXDCQIQVMZJXUZ346MBUPUESCCQJY53EQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.455787740 (528h27m39.326870785s) [noise=true , address=145.79.227.3:4150 , piece_id=QK4YPFONOJ5LYLT76JO3LXBFHEMMTZ65VR66QUX4LFG4NC3DQ7HQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.456260731 (528h27m39.327343776s) [noise=true , address=145.79.227.15:4874 , piece_id=LCOSGDSBU5XA4Y24ITKZ2YEJ7Q62UOUKXIZHPH2AZGTQSDJNV5TA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.461398053 (528h27m39.332481098s) [noise=true , address=145.79.227.52:4445 , piece_id=XAR7FXE52GRZND4LAD5QQ76UCAA3R2WWPXWF3U77K6S4QQH5CU4A, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.464695132 (528h27m39.335778167s) [noise=true , address=145.79.227.17:4970 , piece_id=YN73TI6N63IBCVV5PGJA23PNPC4ZVXWAO65GRJZZP4OK45MFO7LQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.474375355 (528h27m39.34545839s) [noise=true , address=145.79.227.13:4804 , piece_id=TKOD47CAYP66ZMQHVZH7BJCZ2D3HT7QQI3MUMNAXOBK7AAGP3IIQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.484112898 (528h27m39.355195933s) [noise=true , address=145.79.227.18:4007 , piece_id=RWNT2X7NGRDJQ7O34YNQNJVQVXES7UZMSWGAX7LUXVXIWMRYIBXA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.618260909 (528h27m39.489343954s) [noise=true , address=77.90.31.11:4554 , piece_id=7TENHVIOP5HIYH6X3NHHUFRUVIP3YTDLAI6NA7DCEWG4GTZUZYCA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.711842235 (528h27m39.58292528s) [noise=true , address=45.91.116.10:4081 , piece_id=AEO2DIWH44M4POUNXVJXT7PQG5QI4IOFAQRWROTV5ICZTJW72NDA, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.795897832 (528h27m39.666980877s) [noise=true , address=45.91.116.6:4974 , piece_id=O3E3XHKWWVW7OH7KB4T3IR6QA34HXSPRYSBAGP2YALGSYQTQESQQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: <nil> Dec 27 15:56:39.795923310 (528h27m39.667006355s) [seg_pos={0 0}, upload=463507]: Segment reached optimal threshold of 13 pieces. Dec 27 15:56:39.796163628 (528h27m39.667246673s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 15:56:39.796172264 (528h27m39.667255299s) [noise=true , address=45.91.116.13:4154 , piece_id=PPNNATAN2IQWLG6DDUZJXDPI35AEBSBBVLOPAMPCDW5YGVFDUKYQ, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796350234 (528h27m39.667433279s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 15:56:39.796366124 (528h27m39.667449169s) [noise=true , address=77.90.31.21:4877 , piece_id=VBDMPVIZIXRU5RIZYI2S36D4I5NVIKL36MLGHSJSLNHVQHOSLS4Q, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796487524 (528h27m39.667570569s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34912->45.91.116.5:4297: use of closed network connection Dec 27 15:56:39.796497454 (528h27m39.667580499s) [noise=true , address=45.91.116.5:4297 , piece_id=W3ECDECNFNM66QSUUXYZVELDP463DJB2YLL4T5OD33APVITPCZRQ, seg_pos={0 0}, upload=463507]: Long tail cancellation due to optimal threshold Dec 27 15:56:39.796518253 (528h27m39.667601298s) [seg_pos={0 0}, upload=463507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:56:39.796617292 (528h27m39.667700337s) [upload=463507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:56:39.796624415 (528h27m39.66770745s) [upload=463507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:56:40.092401729 (528h27m39.963484774s) [upload=463507]: Done uploading segments... Dec 27 15:56:40.092421216 (528h27m39.963504261s) [upload=463507]: Upload finished. err: <nil> Duration: 30.003187988s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X...] duration: 4.868753771s trace: https://jaeger.dev.storj.io/trace/365412d41f860e21 FAIL (2025-12-27 15:56:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.86877983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X.....XX......X................XX...........] duration: 50.698866602s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:56:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 50.698898152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.X.....X...X......X.X..X..XXX...........X.......X..X.XX..] duration: 5.684016487s trace: https://jaeger.dev.storj.io/trace/62a41496661bfd24 FAIL (2025-12-27 15:55:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA) context deadline exceeded ======LOGS====== d=463476]: Done starting upload Dec 27 15:54:35.076369259 (528h25m34.947452304s) [upload=463476]: Uploading segments... Dec 27 15:54:35.076600400 (528h25m34.947683445s) [upload=463476]: Got next segment. Inline: false Dec 27 15:54:35.076647379 (528h25m34.947730424s) [upload=463476]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:54:35.076654473 (528h25m34.947737508s) [upload=463476]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:54:35.327535956 (528h25m35.198619001s) [seg_pos={0 0}, upload=463476]: Begin upload segment... Dec 27 15:54:35.327591532 (528h25m35.198674567s) [seg_pos={0 0}, upload=463476]: Done begin upload segment. Dec 27 15:54:35.327684509 (528h25m35.198767544s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.327693436 (528h25m35.198776491s) [noise=true , address=145.79.227.38:4978 , piece_id=LK6LEFUU4WTSRERTGJOT5VR5OMROAXVDIXQRKSR6KJ4Y45MUYEBA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.328367259 (528h25m35.199450304s) [noise=true , address=145.79.227.7:4295 , piece_id=F26NOYCQ44U6R4AITZUXJ2IOUTGUEDF2HGCS5MO4X7DVQGX7NZMA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.329112479 (528h25m35.200195524s) [noise=true , address=145.79.227.15:4874 , piece_id=GXIO3AOGAMVJ7YIEMC2HGW47JM7VBW2VI22GHI5MIYYI2J375ULA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.329562265 (528h25m35.2006453s) [noise=true , address=145.79.227.47:4267 , piece_id=BCAVVM6JYNAQGI7RSG5K7LZPZ6FJGQJFCORZDFRL6RWVE7AB4SXA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330011641 (528h25m35.201094686s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330193056 (528h25m35.201276091s) [noise=true , address=45.91.116.10:4081 , piece_id=CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330481445 (528h25m35.20156448s) [noise=true , address=145.79.227.16:4910 , piece_id=MA24XNSBILVFQ7WYXULTMFNPZRQBPOBMQXBF7OKBRL65DKZH7TFA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330679823 (528h25m35.201762858s) [noise=true , address=145.79.227.37:4565 , piece_id=UES2F33RM6W6SRCXAF4DTBMUE4OLYYP4FBKYRDRPO5HH2H276FMQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331317066 (528h25m35.202400101s) [noise=true , address=77.90.31.10:4520 , piece_id=AWHIGADFSLZCV75O62WSQO4B7PER7SRYI7ZNDKGJXYKP6FJHW36Q, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331413089 (528h25m35.202496124s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331824884 (528h25m35.202907919s) [noise=true , address=145.79.227.3:4151 , piece_id=4BC7EAK2IAV4BZ7ZY5K2PQBMOCK5UAP4XTIBHSFOZSETTWYIQUTQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.334871555 (528h25m35.205954591s) [noise=true , address=45.91.116.11:4119 , piece_id=J4WU6327QZYHFNY7KFTQOUJ56D6WYMPRJWVW6HIMAFZY2GLB2CDA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.335458283 (528h25m35.206541328s) [noise=true , address=145.79.227.13:4802 , piece_id=ZPSCOAYHAYD2GWUCDSQMFWGBGFSBQHCLBNIBP5OJ5FXSB6NSRIHQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.336059958 (528h25m35.207143003s) [noise=true , address=145.79.227.12:4765 , piece_id=4H74FQ4EJ5EYRTG4HMCBFNQEGRJBXJDLGN3MJKDEUOFLKRM2KLPA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.336823362 (528h25m35.207906397s) [noise=true , address=45.91.116.9:4264 , piece_id=ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:36.953226693 (528h25m36.824309738s) [upload=463476]: Next returned nil segment Dec 27 15:54:36.953239318 (528h25m36.824322363s) [upload=463476]: Waiting for error group managing segments... Dec 27 15:54:37.339362496 (528h25m37.210445531s) [noise=true , address=145.79.227.37:4565 , piece_id=UES2F33RM6W6SRCXAF4DTBMUE4OLYYP4FBKYRDRPO5HH2H276FMQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.339525657 (528h25m37.210608692s) [noise=true , address=145.79.227.15:4874 , piece_id=GXIO3AOGAMVJ7YIEMC2HGW47JM7VBW2VI22GHI5MIYYI2J375ULA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.339888177 (528h25m37.210971212s) [noise=true , address=145.79.227.16:4910 , piece_id=MA24XNSBILVFQ7WYXULTMFNPZRQBPOBMQXBF7OKBRL65DKZH7TFA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.340694682 (528h25m37.211777727s) [noise=true , address=145.79.227.7:4295 , piece_id=F26NOYCQ44U6R4AITZUXJ2IOUTGUEDF2HGCS5MO4X7DVQGX7NZMA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.340826133 (528h25m37.211909168s) [noise=true , address=145.79.227.12:4765 , piece_id=4H74FQ4EJ5EYRTG4HMCBFNQEGRJBXJDLGN3MJKDEUOFLKRM2KLPA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.351814256 (528h25m37.222897291s) [noise=true , address=145.79.227.38:4978 , piece_id=LK6LEFUU4WTSRERTGJOT5VR5OMROAXVDIXQRKSR6KJ4Y45MUYEBA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.355404823 (528h25m37.226487858s) [noise=true , address=145.79.227.47:4267 , piece_id=BCAVVM6JYNAQGI7RSG5K7LZPZ6FJGQJFCORZDFRL6RWVE7AB4SXA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.361828895 (528h25m37.23291194s) [noise=true , address=145.79.227.13:4802 , piece_id=ZPSCOAYHAYD2GWUCDSQMFWGBGFSBQHCLBNIBP5OJ5FXSB6NSRIHQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.366182255 (528h25m37.23726529s) [noise=true , address=145.79.227.3:4151 , piece_id=4BC7EAK2IAV4BZ7ZY5K2PQBMOCK5UAP4XTIBHSFOZSETTWYIQUTQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.589526383 (528h25m37.460609428s) [noise=true , address=45.91.116.9:4264 , piece_id=ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.591289721 (528h25m37.462372766s) [noise=true , address=45.91.116.11:4119 , piece_id=J4WU6327QZYHFNY7KFTQOUJ56D6WYMPRJWVW6HIMAFZY2GLB2CDA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.136344462 (528h25m38.007427507s) [noise=true , address=45.91.116.10:4081 , piece_id=CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.632331912 (528h25m38.503414947s) [noise=true , address=77.90.31.10:4520 , piece_id=AWHIGADFSLZCV75O62WSQO4B7PER7SRYI7ZNDKGJXYKP6FJHW36Q, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.632388609 (528h25m38.503471654s) [seg_pos={0 0}, upload=463476]: Segment reached optimal threshold of 13 pieces. Dec 27 15:54:38.632689873 (528h25m38.503772918s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 15:54:38.632700243 (528h25m38.503783278s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.632809270 (528h25m38.503892315s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection Dec 27 15:54:38.632828798 (528h25m38.503911843s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.633136925 (528h25m38.50421998s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:47704->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:47704->145.79.227.21:4474: use of closed network connection Dec 27 15:54:38.633147915 (528h25m38.50423095s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.633173063 (528h25m38.504256098s) [seg_pos={0 0}, upload=463476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:54:38.633256392 (528h25m38.504339437s) [upload=463476]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:54:38.633263204 (528h25m38.504346249s) [upload=463476]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:54:38.939937848 (528h25m38.811020893s) [upload=463476]: Done uploading segments... Dec 27 15:54:38.939963287 (528h25m38.811046332s) [upload=463476]: Upload finished. err: <nil> Duration: 5.684049612s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XX...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X..] duration: 5.001406386s trace: https://jaeger.dev.storj.io/trace/365412d41f860e21 FAIL (2025-12-27 15:55:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001463055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X.....XX......X................XX..........] duration: 44.965723911s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:55:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 44.965742745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X.X.....X...X......X.X..X..XXX...........X.......X..X.XX.] duration: 5.98656313s trace: https://jaeger.dev.storj.io/trace/62a41496661bfd24 FAIL (2025-12-27 15:55:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA) context deadline exceeded ======LOGS====== d=463476]: Done starting upload Dec 27 15:54:35.076369259 (528h25m34.947452304s) [upload=463476]: Uploading segments... Dec 27 15:54:35.076600400 (528h25m34.947683445s) [upload=463476]: Got next segment. Inline: false Dec 27 15:54:35.076647379 (528h25m34.947730424s) [upload=463476]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:54:35.076654473 (528h25m34.947737508s) [upload=463476]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:54:35.327535956 (528h25m35.198619001s) [seg_pos={0 0}, upload=463476]: Begin upload segment... Dec 27 15:54:35.327591532 (528h25m35.198674567s) [seg_pos={0 0}, upload=463476]: Done begin upload segment. Dec 27 15:54:35.327684509 (528h25m35.198767544s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.327693436 (528h25m35.198776491s) [noise=true , address=145.79.227.38:4978 , piece_id=LK6LEFUU4WTSRERTGJOT5VR5OMROAXVDIXQRKSR6KJ4Y45MUYEBA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.328367259 (528h25m35.199450304s) [noise=true , address=145.79.227.7:4295 , piece_id=F26NOYCQ44U6R4AITZUXJ2IOUTGUEDF2HGCS5MO4X7DVQGX7NZMA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.329112479 (528h25m35.200195524s) [noise=true , address=145.79.227.15:4874 , piece_id=GXIO3AOGAMVJ7YIEMC2HGW47JM7VBW2VI22GHI5MIYYI2J375ULA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.329562265 (528h25m35.2006453s) [noise=true , address=145.79.227.47:4267 , piece_id=BCAVVM6JYNAQGI7RSG5K7LZPZ6FJGQJFCORZDFRL6RWVE7AB4SXA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330011641 (528h25m35.201094686s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330193056 (528h25m35.201276091s) [noise=true , address=45.91.116.10:4081 , piece_id=CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330481445 (528h25m35.20156448s) [noise=true , address=145.79.227.16:4910 , piece_id=MA24XNSBILVFQ7WYXULTMFNPZRQBPOBMQXBF7OKBRL65DKZH7TFA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330679823 (528h25m35.201762858s) [noise=true , address=145.79.227.37:4565 , piece_id=UES2F33RM6W6SRCXAF4DTBMUE4OLYYP4FBKYRDRPO5HH2H276FMQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331317066 (528h25m35.202400101s) [noise=true , address=77.90.31.10:4520 , piece_id=AWHIGADFSLZCV75O62WSQO4B7PER7SRYI7ZNDKGJXYKP6FJHW36Q, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331413089 (528h25m35.202496124s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331824884 (528h25m35.202907919s) [noise=true , address=145.79.227.3:4151 , piece_id=4BC7EAK2IAV4BZ7ZY5K2PQBMOCK5UAP4XTIBHSFOZSETTWYIQUTQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.334871555 (528h25m35.205954591s) [noise=true , address=45.91.116.11:4119 , piece_id=J4WU6327QZYHFNY7KFTQOUJ56D6WYMPRJWVW6HIMAFZY2GLB2CDA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.335458283 (528h25m35.206541328s) [noise=true , address=145.79.227.13:4802 , piece_id=ZPSCOAYHAYD2GWUCDSQMFWGBGFSBQHCLBNIBP5OJ5FXSB6NSRIHQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.336059958 (528h25m35.207143003s) [noise=true , address=145.79.227.12:4765 , piece_id=4H74FQ4EJ5EYRTG4HMCBFNQEGRJBXJDLGN3MJKDEUOFLKRM2KLPA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.336823362 (528h25m35.207906397s) [noise=true , address=45.91.116.9:4264 , piece_id=ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:36.953226693 (528h25m36.824309738s) [upload=463476]: Next returned nil segment Dec 27 15:54:36.953239318 (528h25m36.824322363s) [upload=463476]: Waiting for error group managing segments... Dec 27 15:54:37.339362496 (528h25m37.210445531s) [noise=true , address=145.79.227.37:4565 , piece_id=UES2F33RM6W6SRCXAF4DTBMUE4OLYYP4FBKYRDRPO5HH2H276FMQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.339525657 (528h25m37.210608692s) [noise=true , address=145.79.227.15:4874 , piece_id=GXIO3AOGAMVJ7YIEMC2HGW47JM7VBW2VI22GHI5MIYYI2J375ULA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.339888177 (528h25m37.210971212s) [noise=true , address=145.79.227.16:4910 , piece_id=MA24XNSBILVFQ7WYXULTMFNPZRQBPOBMQXBF7OKBRL65DKZH7TFA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.340694682 (528h25m37.211777727s) [noise=true , address=145.79.227.7:4295 , piece_id=F26NOYCQ44U6R4AITZUXJ2IOUTGUEDF2HGCS5MO4X7DVQGX7NZMA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.340826133 (528h25m37.211909168s) [noise=true , address=145.79.227.12:4765 , piece_id=4H74FQ4EJ5EYRTG4HMCBFNQEGRJBXJDLGN3MJKDEUOFLKRM2KLPA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.351814256 (528h25m37.222897291s) [noise=true , address=145.79.227.38:4978 , piece_id=LK6LEFUU4WTSRERTGJOT5VR5OMROAXVDIXQRKSR6KJ4Y45MUYEBA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.355404823 (528h25m37.226487858s) [noise=true , address=145.79.227.47:4267 , piece_id=BCAVVM6JYNAQGI7RSG5K7LZPZ6FJGQJFCORZDFRL6RWVE7AB4SXA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.361828895 (528h25m37.23291194s) [noise=true , address=145.79.227.13:4802 , piece_id=ZPSCOAYHAYD2GWUCDSQMFWGBGFSBQHCLBNIBP5OJ5FXSB6NSRIHQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.366182255 (528h25m37.23726529s) [noise=true , address=145.79.227.3:4151 , piece_id=4BC7EAK2IAV4BZ7ZY5K2PQBMOCK5UAP4XTIBHSFOZSETTWYIQUTQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.589526383 (528h25m37.460609428s) [noise=true , address=45.91.116.9:4264 , piece_id=ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.591289721 (528h25m37.462372766s) [noise=true , address=45.91.116.11:4119 , piece_id=J4WU6327QZYHFNY7KFTQOUJ56D6WYMPRJWVW6HIMAFZY2GLB2CDA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.136344462 (528h25m38.007427507s) [noise=true , address=45.91.116.10:4081 , piece_id=CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.632331912 (528h25m38.503414947s) [noise=true , address=77.90.31.10:4520 , piece_id=AWHIGADFSLZCV75O62WSQO4B7PER7SRYI7ZNDKGJXYKP6FJHW36Q, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.632388609 (528h25m38.503471654s) [seg_pos={0 0}, upload=463476]: Segment reached optimal threshold of 13 pieces. Dec 27 15:54:38.632689873 (528h25m38.503772918s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 15:54:38.632700243 (528h25m38.503783278s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.632809270 (528h25m38.503892315s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection Dec 27 15:54:38.632828798 (528h25m38.503911843s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.633136925 (528h25m38.50421998s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:47704->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:47704->145.79.227.21:4474: use of closed network connection Dec 27 15:54:38.633147915 (528h25m38.50423095s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.633173063 (528h25m38.504256098s) [seg_pos={0 0}, upload=463476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:54:38.633256392 (528h25m38.504339437s) [upload=463476]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:54:38.633263204 (528h25m38.504346249s) [upload=463476]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:54:38.939937848 (528h25m38.811020893s) [upload=463476]: Done uploading segments... Dec 27 15:54:38.939963287 (528h25m38.811046332s) [upload=463476]: Upload finished. err: <nil> Duration: 5.986584142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..X.X.....X...X......X.X..X..XXX...........X.......X..X.XX] duration: 30.006818821s trace: https://jaeger.dev.storj.io/trace/62a41496661bfd24 FAIL (2025-12-27 15:55:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA) context deadline exceeded ======LOGS====== d=463476]: Done starting upload Dec 27 15:54:35.076369259 (528h25m34.947452304s) [upload=463476]: Uploading segments... Dec 27 15:54:35.076600400 (528h25m34.947683445s) [upload=463476]: Got next segment. Inline: false Dec 27 15:54:35.076647379 (528h25m34.947730424s) [upload=463476]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:54:35.076654473 (528h25m34.947737508s) [upload=463476]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:54:35.327535956 (528h25m35.198619001s) [seg_pos={0 0}, upload=463476]: Begin upload segment... Dec 27 15:54:35.327591532 (528h25m35.198674567s) [seg_pos={0 0}, upload=463476]: Done begin upload segment. Dec 27 15:54:35.327684509 (528h25m35.198767544s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.327693436 (528h25m35.198776491s) [noise=true , address=145.79.227.38:4978 , piece_id=LK6LEFUU4WTSRERTGJOT5VR5OMROAXVDIXQRKSR6KJ4Y45MUYEBA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.328367259 (528h25m35.199450304s) [noise=true , address=145.79.227.7:4295 , piece_id=F26NOYCQ44U6R4AITZUXJ2IOUTGUEDF2HGCS5MO4X7DVQGX7NZMA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.329112479 (528h25m35.200195524s) [noise=true , address=145.79.227.15:4874 , piece_id=GXIO3AOGAMVJ7YIEMC2HGW47JM7VBW2VI22GHI5MIYYI2J375ULA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.329562265 (528h25m35.2006453s) [noise=true , address=145.79.227.47:4267 , piece_id=BCAVVM6JYNAQGI7RSG5K7LZPZ6FJGQJFCORZDFRL6RWVE7AB4SXA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330011641 (528h25m35.201094686s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330193056 (528h25m35.201276091s) [noise=true , address=45.91.116.10:4081 , piece_id=CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330481445 (528h25m35.20156448s) [noise=true , address=145.79.227.16:4910 , piece_id=MA24XNSBILVFQ7WYXULTMFNPZRQBPOBMQXBF7OKBRL65DKZH7TFA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.330679823 (528h25m35.201762858s) [noise=true , address=145.79.227.37:4565 , piece_id=UES2F33RM6W6SRCXAF4DTBMUE4OLYYP4FBKYRDRPO5HH2H276FMQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331317066 (528h25m35.202400101s) [noise=true , address=77.90.31.10:4520 , piece_id=AWHIGADFSLZCV75O62WSQO4B7PER7SRYI7ZNDKGJXYKP6FJHW36Q, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331413089 (528h25m35.202496124s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.331824884 (528h25m35.202907919s) [noise=true , address=145.79.227.3:4151 , piece_id=4BC7EAK2IAV4BZ7ZY5K2PQBMOCK5UAP4XTIBHSFOZSETTWYIQUTQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.334871555 (528h25m35.205954591s) [noise=true , address=45.91.116.11:4119 , piece_id=J4WU6327QZYHFNY7KFTQOUJ56D6WYMPRJWVW6HIMAFZY2GLB2CDA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.335458283 (528h25m35.206541328s) [noise=true , address=145.79.227.13:4802 , piece_id=ZPSCOAYHAYD2GWUCDSQMFWGBGFSBQHCLBNIBP5OJ5FXSB6NSRIHQ, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.336059958 (528h25m35.207143003s) [noise=true , address=145.79.227.12:4765 , piece_id=4H74FQ4EJ5EYRTG4HMCBFNQEGRJBXJDLGN3MJKDEUOFLKRM2KLPA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:35.336823362 (528h25m35.207906397s) [noise=true , address=45.91.116.9:4264 , piece_id=ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA, seg_pos={0 0}, upload=463476]: Uploading piece... Dec 27 15:54:36.953226693 (528h25m36.824309738s) [upload=463476]: Next returned nil segment Dec 27 15:54:36.953239318 (528h25m36.824322363s) [upload=463476]: Waiting for error group managing segments... Dec 27 15:54:37.339362496 (528h25m37.210445531s) [noise=true , address=145.79.227.37:4565 , piece_id=UES2F33RM6W6SRCXAF4DTBMUE4OLYYP4FBKYRDRPO5HH2H276FMQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.339525657 (528h25m37.210608692s) [noise=true , address=145.79.227.15:4874 , piece_id=GXIO3AOGAMVJ7YIEMC2HGW47JM7VBW2VI22GHI5MIYYI2J375ULA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.339888177 (528h25m37.210971212s) [noise=true , address=145.79.227.16:4910 , piece_id=MA24XNSBILVFQ7WYXULTMFNPZRQBPOBMQXBF7OKBRL65DKZH7TFA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.340694682 (528h25m37.211777727s) [noise=true , address=145.79.227.7:4295 , piece_id=F26NOYCQ44U6R4AITZUXJ2IOUTGUEDF2HGCS5MO4X7DVQGX7NZMA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.340826133 (528h25m37.211909168s) [noise=true , address=145.79.227.12:4765 , piece_id=4H74FQ4EJ5EYRTG4HMCBFNQEGRJBXJDLGN3MJKDEUOFLKRM2KLPA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.351814256 (528h25m37.222897291s) [noise=true , address=145.79.227.38:4978 , piece_id=LK6LEFUU4WTSRERTGJOT5VR5OMROAXVDIXQRKSR6KJ4Y45MUYEBA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.355404823 (528h25m37.226487858s) [noise=true , address=145.79.227.47:4267 , piece_id=BCAVVM6JYNAQGI7RSG5K7LZPZ6FJGQJFCORZDFRL6RWVE7AB4SXA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.361828895 (528h25m37.23291194s) [noise=true , address=145.79.227.13:4802 , piece_id=ZPSCOAYHAYD2GWUCDSQMFWGBGFSBQHCLBNIBP5OJ5FXSB6NSRIHQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.366182255 (528h25m37.23726529s) [noise=true , address=145.79.227.3:4151 , piece_id=4BC7EAK2IAV4BZ7ZY5K2PQBMOCK5UAP4XTIBHSFOZSETTWYIQUTQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.589526383 (528h25m37.460609428s) [noise=true , address=45.91.116.9:4264 , piece_id=ABLW6CO5WEE7EPF5GA2R34OXGRM3VZTSQMB6XUG72GELLP7B7EVA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:37.591289721 (528h25m37.462372766s) [noise=true , address=45.91.116.11:4119 , piece_id=J4WU6327QZYHFNY7KFTQOUJ56D6WYMPRJWVW6HIMAFZY2GLB2CDA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.136344462 (528h25m38.007427507s) [noise=true , address=45.91.116.10:4081 , piece_id=CZJRL75QM5B4D7ZH4VYFDA32B6AJWPVPJMIOMQLMR2ORNLIVNEFA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.632331912 (528h25m38.503414947s) [noise=true , address=77.90.31.10:4520 , piece_id=AWHIGADFSLZCV75O62WSQO4B7PER7SRYI7ZNDKGJXYKP6FJHW36Q, seg_pos={0 0}, upload=463476]: Done uploading piece. err: <nil> Dec 27 15:54:38.632388609 (528h25m38.503471654s) [seg_pos={0 0}, upload=463476]: Segment reached optimal threshold of 13 pieces. Dec 27 15:54:38.632689873 (528h25m38.503772918s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 15:54:38.632700243 (528h25m38.503783278s) [noise=true , address=45.91.116.4:4939 , piece_id=VBZZVTOVB5T2WGENOKSSEHU7XWMDWHPR2QNK5QOUGPS6UOX5HLYQ, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.632809270 (528h25m38.503892315s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54674->145.79.227.29:4941: use of closed network connection Dec 27 15:54:38.632828798 (528h25m38.503911843s) [noise=true , address=145.79.227.29:4941 , piece_id=EEWS5U4UF5IN4XQMJ4PJV2GOAFH6ZTN5B2RB4PQLKM62CFOOJ4EA, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.633136925 (528h25m38.50421998s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:47704->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:47704->145.79.227.21:4474: use of closed network connection Dec 27 15:54:38.633147915 (528h25m38.50423095s) [noise=true , address=145.79.227.21:4474 , piece_id=RX2CT5N7XY4FVTHIJUM5RT7OLT55ODTKEWVAUGKKYMXJDTNZSDYQ, seg_pos={0 0}, upload=463476]: Long tail cancellation due to optimal threshold Dec 27 15:54:38.633173063 (528h25m38.504256098s) [seg_pos={0 0}, upload=463476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:54:38.633256392 (528h25m38.504339437s) [upload=463476]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:54:38.633263204 (528h25m38.504346249s) [upload=463476]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:54:38.939937848 (528h25m38.811020893s) [upload=463476]: Done uploading segments... Dec 27 15:54:38.939963287 (528h25m38.811046332s) [upload=463476]: Upload finished. err: <nil> Duration: 30.006836745s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XXX...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X.] duration: 3.731634963s trace: https://jaeger.dev.storj.io/trace/365412d41f860e21 FAIL (2025-12-27 15:54:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.731656514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X.....XX......X................XX.........] duration: 52.782072413s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:54:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 52.782099042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXXX...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......X] duration: 5.001722465s trace: https://jaeger.dev.storj.io/trace/365412d41f860e21 FAIL (2025-12-27 15:54:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001745379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X..X.X.....X...X......X.X..X..XXX...........X.......X..X.X] duration: 30.005513493s trace: https://jaeger.dev.storj.io/trace/58ff46e5f866c911 FAIL (2025-12-27 15:53:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: L75TMUE7ZACMLSVMLJXIIPR3UG3XE2TWWMUUJ2W52JHV5OTEFFDA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ABSSYZUM5GSTSNL2SDS7B4DFDCYFBBQSKOA5YYJ56WLMMWJDFCCQ) context deadline exceeded ======LOGS====== (528h24m29.046572633s) [upload=463461]: Done starting upload Dec 27 15:53:29.175583848 (528h24m29.046666893s) [upload=463461]: Uploading segments... Dec 27 15:53:29.175854674 (528h24m29.046937719s) [upload=463461]: Got next segment. Inline: false Dec 27 15:53:29.175907795 (528h24m29.04699084s) [upload=463461]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:53:29.175933434 (528h24m29.047016469s) [upload=463461]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:53:29.425093157 (528h24m29.296176192s) [seg_pos={0 0}, upload=463461]: Begin upload segment... Dec 27 15:53:29.425159774 (528h24m29.296242809s) [seg_pos={0 0}, upload=463461]: Done begin upload segment. Dec 27 15:53:29.425191795 (528h24m29.29627483s) [noise=true , address=77.90.31.16:4736 , piece_id=42LF2PTZGU6PTNJJ4G7UA7OPXTZ2J6BO2LP25KKL67TB2WCQ7TDQ, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.425304981 (528h24m29.296388016s) [noise=true , address=45.91.116.14:4191 , piece_id=55GOQBNK3ZKX4C7ZD5BVAGLOR7R3TRUFJ3KHE4IHTTTESOYLENCQ, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.425752232 (528h24m29.296835267s) [noise=true , address=145.79.227.51:4410 , piece_id=6TGRHB5P5NANC4UKEB2YBJRRHXD4ZZDI6NGK4XCHX5OQYIPLCKVA, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.426157644 (528h24m29.297240679s) [noise=true , address=45.91.116.13:4154 , piece_id=L75TMUE7ZACMLSVMLJXIIPR3UG3XE2TWWMUUJ2W52JHV5OTEFFDA, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.426461703 (528h24m29.297544738s) [noise=true , address=145.79.227.8:4329 , piece_id=CGLXNPH64FZW3S7XPVRJ3ZEYHKPY36WMZOPFLALJ2ZJNWMONGONQ, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.426743129 (528h24m29.297826174s) [noise=true , address=145.79.227.39:4013 , piece_id=2VZJ76HTAHWUFHF6HQ4Z3UZBYTGTV5HSGOY4RFET6RKI3VNTKG2Q, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.426994198 (528h24m29.298077233s) [noise=true , address=145.79.227.47:4265 , piece_id=NE6BJ3GRNSEMKIIA6QKW7SZ4LZLR2CD7AGXNK24FRUFUWQZGBWKQ, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.427237129 (528h24m29.298320164s) [noise=true , address=45.91.116.5:4300 , piece_id=BZNDCPCH2UGLLR6VU5A2J47GFB42EXDBEMQPRH2L6Y7PVJG5SSAQ, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.427501523 (528h24m29.298584568s) [noise=true , address=45.91.116.6:4975 , piece_id=ABSSYZUM5GSTSNL2SDS7B4DFDCYFBBQSKOA5YYJ56WLMMWJDFCCQ, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.430133946 (528h24m29.301216981s) [noise=true , address=145.79.227.15:4876 , piece_id=QPP42CVLT3DZLCYQ4K2FLHDPGG4DEO7YVQNXPCTSWRYCYK4F5QAA, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.430493019 (528h24m29.301576055s) [noise=true , address=145.79.227.42:4121 , piece_id=GWCUVIGG2F662GQK7BNAK2TYUQOKDGGMYK65HZZJFXHTMBEBQ56Q, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.430957864 (528h24m29.302040899s) [noise=true , address=77.90.31.13:4629 , piece_id=QATZNOMSCQSODIZ7AJHP6P5TJMKK3X2CHGQPROCNB6YEZJ3HNGUQ, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.431233139 (528h24m29.302316174s) [noise=true , address=145.79.227.4:4186 , piece_id=WU2A6ZDX46GSQEPZU5BBGO6DQPKAV7NLZAZZCQPTDLZ2BEOYYOQA, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.431625717 (528h24m29.302708752s) [noise=true , address=77.90.31.11:4554 , piece_id=DAJTJV772U3EEU6LCAGWPCGVYI2HFNCH3B4JYH2ZOVS3CBJP6I5A, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.431948271 (528h24m29.303031326s) [noise=true , address=45.91.116.4:4940 , piece_id=SFDLR7CKFNPEWWKEM2SW44NKNNJAUT322OUSYMWV2WGY4TPFAYAQ, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:29.432278259 (528h24m29.303361294s) [noise=true , address=145.79.227.45:4229 , piece_id=XMNAJ3ECWDJTKEYRU6GKZ45NK44MCL5Y3AIZXL4YRWDQ3HIR55PA, seg_pos={0 0}, upload=463461]: Uploading piece... Dec 27 15:53:31.143463024 (528h24m31.014546059s) [upload=463461]: Next returned nil segment Dec 27 15:53:31.143475809 (528h24m31.014558854s) [upload=463461]: Waiting for error group managing segments... Dec 27 15:53:31.430927918 (528h24m31.302010953s) [noise=true , address=145.79.227.45:4229 , piece_id=XMNAJ3ECWDJTKEYRU6GKZ45NK44MCL5Y3AIZXL4YRWDQ3HIR55PA, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.434348260 (528h24m31.305431306s) [noise=true , address=145.79.227.51:4410 , piece_id=6TGRHB5P5NANC4UKEB2YBJRRHXD4ZZDI6NGK4XCHX5OQYIPLCKVA, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.435798082 (528h24m31.306881127s) [noise=true , address=145.79.227.4:4186 , piece_id=WU2A6ZDX46GSQEPZU5BBGO6DQPKAV7NLZAZZCQPTDLZ2BEOYYOQA, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.436797635 (528h24m31.30788067s) [noise=true , address=145.79.227.8:4329 , piece_id=CGLXNPH64FZW3S7XPVRJ3ZEYHKPY36WMZOPFLALJ2ZJNWMONGONQ, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.460378346 (528h24m31.331461382s) [noise=true , address=145.79.227.42:4121 , piece_id=GWCUVIGG2F662GQK7BNAK2TYUQOKDGGMYK65HZZJFXHTMBEBQ56Q, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.461294141 (528h24m31.332377186s) [noise=true , address=145.79.227.15:4876 , piece_id=QPP42CVLT3DZLCYQ4K2FLHDPGG4DEO7YVQNXPCTSWRYCYK4F5QAA, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.462712190 (528h24m31.333795226s) [noise=true , address=145.79.227.39:4013 , piece_id=2VZJ76HTAHWUFHF6HQ4Z3UZBYTGTV5HSGOY4RFET6RKI3VNTKG2Q, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.466742856 (528h24m31.337825901s) [noise=true , address=145.79.227.47:4265 , piece_id=NE6BJ3GRNSEMKIIA6QKW7SZ4LZLR2CD7AGXNK24FRUFUWQZGBWKQ, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.676540437 (528h24m31.547623472s) [noise=true , address=45.91.116.13:4154 , piece_id=L75TMUE7ZACMLSVMLJXIIPR3UG3XE2TWWMUUJ2W52JHV5OTEFFDA, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.710276556 (528h24m31.581359601s) [noise=true , address=77.90.31.11:4554 , piece_id=DAJTJV772U3EEU6LCAGWPCGVYI2HFNCH3B4JYH2ZOVS3CBJP6I5A, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:31.814508292 (528h24m31.685591337s) [noise=true , address=45.91.116.5:4300 , piece_id=BZNDCPCH2UGLLR6VU5A2J47GFB42EXDBEMQPRH2L6Y7PVJG5SSAQ, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:32.578132496 (528h24m32.449215541s) [noise=true , address=45.91.116.14:4191 , piece_id=55GOQBNK3ZKX4C7ZD5BVAGLOR7R3TRUFJ3KHE4IHTTTESOYLENCQ, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:32.579027180 (528h24m32.450110225s) [noise=true , address=45.91.116.6:4975 , piece_id=ABSSYZUM5GSTSNL2SDS7B4DFDCYFBBQSKOA5YYJ56WLMMWJDFCCQ, seg_pos={0 0}, upload=463461]: Done uploading piece. err: <nil> Dec 27 15:53:32.579050725 (528h24m32.45013377s) [seg_pos={0 0}, upload=463461]: Segment reached optimal threshold of 13 pieces. Dec 27 15:53:32.579361126 (528h24m32.450444171s) [noise=true , address=77.90.31.13:4629 , piece_id=QATZNOMSCQSODIZ7AJHP6P5TJMKK3X2CHGQPROCNB6YEZJ3HNGUQ, seg_pos={0 0}, upload=463461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:46174->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:46174->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46174->77.90.31.13:4629: use of closed network connection Dec 27 15:53:32.579374431 (528h24m32.450457476s) [noise=true , address=77.90.31.13:4629 , piece_id=QATZNOMSCQSODIZ7AJHP6P5TJMKK3X2CHGQPROCNB6YEZJ3HNGUQ, seg_pos={0 0}, upload=463461]: Long tail cancellation due to optimal threshold Dec 27 15:53:32.579441168 (528h24m32.450524203s) [noise=true , address=45.91.116.4:4940 , piece_id=SFDLR7CKFNPEWWKEM2SW44NKNNJAUT322OUSYMWV2WGY4TPFAYAQ, seg_pos={0 0}, upload=463461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 27 15:53:32.579455555 (528h24m32.45053859s) [noise=true , address=45.91.116.4:4940 , piece_id=SFDLR7CKFNPEWWKEM2SW44NKNNJAUT322OUSYMWV2WGY4TPFAYAQ, seg_pos={0 0}, upload=463461]: Long tail cancellation due to optimal threshold Dec 27 15:53:32.579987298 (528h24m32.451070334s) [noise=true , address=77.90.31.16:4736 , piece_id=42LF2PTZGU6PTNJJ4G7UA7OPXTZ2J6BO2LP25KKL67TB2WCQ7TDQ, seg_pos={0 0}, upload=463461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:40636->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:53:32.580002056 (528h24m32.451085111s) [noise=true , address=77.90.31.16:4736 , piece_id=42LF2PTZGU6PTNJJ4G7UA7OPXTZ2J6BO2LP25KKL67TB2WCQ7TDQ, seg_pos={0 0}, upload=463461]: Long tail cancellation due to optimal threshold Dec 27 15:53:32.580042343 (528h24m32.451125398s) [seg_pos={0 0}, upload=463461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:53:32.580166449 (528h24m32.451249495s) [upload=463461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:53:32.580172912 (528h24m32.451255947s) [upload=463461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:53:32.867643436 (528h24m32.738726491s) [upload=463461]: Done uploading segments... Dec 27 15:53:32.867709532 (528h24m32.738792567s) [upload=463461]: Upload finished. err: <nil> Duration: 30.005534702s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [.XXXX...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX......] duration: 3.504323767s trace: https://jaeger.dev.storj.io/trace/534a269c5f8f3f97 FAIL (2025-12-27 15:53:46 UTC) orbiter warning: context deadline exceeded Duration: 3.504351208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X..X.X.....X...X......X.X..X..XXX...........X.......X..X.] duration: 5.625087603s trace: https://jaeger.dev.storj.io/trace/7cf60a64b935d292 FAIL (2025-12-27 15:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TR7P4KFGCQTOLTJ45PGKOAQZ5I45XJ5NPYHGCLEFYFVET5TGDQVA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 7L7U3VR63WOGY4LDT5E3CYNTGADMRBNXX4RNYHSZ77MQSGR34AMQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: P7LT6TGLWHV7RUR3CL2532TKOB2XZHGHN26V7CMULNXA2FDGHOGA) context deadline exceeded ======LOGS====== Dec 27 15:52:41.366331860 (528h23m41.237414895s) [upload=463448]: Starting upload Dec 27 15:52:41.366391583 (528h23m41.237474618s) [upload=463448]: Done starting upload Dec 27 15:52:41.366498497 (528h23m41.237581542s) [upload=463448]: Uploading segments... Dec 27 15:52:41.366679592 (528h23m41.237762627s) [upload=463448]: Got next segment. Inline: false Dec 27 15:52:41.366754415 (528h23m41.23783746s) [upload=463448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:52:41.366760496 (528h23m41.237843541s) [upload=463448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:52:41.617621119 (528h23m41.488704164s) [seg_pos={0 0}, upload=463448]: Begin upload segment... Dec 27 15:52:41.617848743 (528h23m41.488931788s) [seg_pos={0 0}, upload=463448]: Done begin upload segment. Dec 27 15:52:41.617907554 (528h23m41.488990599s) [noise=true , address=145.79.227.11:4730 , piece_id=IYFN7VTLD6I4Y2UUL2Q3URQSNEGB37XEYIEAYEQRZL6VRECZZHUQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.617926300 (528h23m41.489009345s) [noise=true , address=145.79.227.51:4409 , piece_id=LNMYX6DXKV6OYNBGK4D4AMIPPXXUFADUUKYH4N3Y6ZNYWV4PZVWA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.618600894 (528h23m41.489683939s) [noise=true , address=145.79.227.49:4339 , piece_id=5PAETDZASQLO47SSXBHLLTFYXORQS27WKIIAMHRVWISO6FHRORXA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.618709451 (528h23m41.489792496s) [noise=true , address=45.91.116.4:4939 , piece_id=PRZXTLWPCI36PFH3J5FPPHZIJPJEHUZIKGZLVKQV5Y7DVXQONSPA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.619061641 (528h23m41.490144676s) [noise=true , address=45.91.116.11:4118 , piece_id=TR7P4KFGCQTOLTJ45PGKOAQZ5I45XJ5NPYHGCLEFYFVET5TGDQVA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.619263416 (528h23m41.490346451s) [noise=true , address=145.79.227.5:4223 , piece_id=2T2YRLBMQKHE6KKKNKAFBHO5EZDZDP6S2CWB3H4IAXVJXMXF5PZQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.619616217 (528h23m41.490699252s) [noise=true , address=77.90.31.11:4556 , piece_id=QSOG73Y6IGJ334UYSHF4HOWNXIQD635C7NJ66UPDS4CSPUQAFTDA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.619758299 (528h23m41.490841344s) [noise=true , address=145.79.227.52:4445 , piece_id=BLM6FGNXUD7HPU5JDRZE6YXZPKF3QTYVXJLEHMUVA6URPGLZGGWA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.620060964 (528h23m41.491143999s) [noise=true , address=77.90.31.18:4808 , piece_id=7L7U3VR63WOGY4LDT5E3CYNTGADMRBNXX4RNYHSZ77MQSGR34AMQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.620274010 (528h23m41.491357045s) [noise=true , address=77.90.31.16:4735 , piece_id=P7LT6TGLWHV7RUR3CL2532TKOB2XZHGHN26V7CMULNXA2FDGHOGA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.620561077 (528h23m41.491644122s) [noise=true , address=45.91.116.13:4154 , piece_id=3AEGYH2XRZ4YJ2I5A7WGR4VPUOQTSQAU4CW4P6ERTO5UMTP6NDTQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.620792708 (528h23m41.491875753s) [noise=true , address=145.79.227.42:4121 , piece_id=6PM7JIBT6G5C43DZK5VOOQCQX2H4P3HNEWCYRTH43D5P3ITSASKQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.621283582 (528h23m41.492366617s) [noise=true , address=45.91.116.5:4298 , piece_id=OXCFVA5L3YKUJOPSSQJ3HAIZ77KDKFLI4B2KLE6BFMTYLKSFU54Q, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.621843879 (528h23m41.492926924s) [noise=true , address=145.79.227.4:4186 , piece_id=2VV7M2LTB3TTBMIHPY5UJVNHHLHCRAGAFBQNW4OBGSQEEES2Y2CA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.622356886 (528h23m41.493439931s) [noise=true , address=145.79.227.48:4301 , piece_id=LDACISJDANBDHKKUKGYWN4FIIDJWTGDGD3SOOE5ZUXZK2XXOLQHQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.623061969 (528h23m41.494145014s) [noise=true , address=145.79.227.10:4402 , piece_id=EPTGZDYB6URVLRJAIUXELVA6JYCXNDA5AEFJ5WWVP77UBNPDLHPQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:43.242171828 (528h23m43.113254863s) [upload=463448]: Next returned nil segment Dec 27 15:52:43.242184562 (528h23m43.113267607s) [upload=463448]: Waiting for error group managing segments... Dec 27 15:52:43.627203516 (528h23m43.498286561s) [noise=true , address=145.79.227.10:4402 , piece_id=EPTGZDYB6URVLRJAIUXELVA6JYCXNDA5AEFJ5WWVP77UBNPDLHPQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.628848718 (528h23m43.499931753s) [noise=true , address=145.79.227.42:4121 , piece_id=6PM7JIBT6G5C43DZK5VOOQCQX2H4P3HNEWCYRTH43D5P3ITSASKQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.629780062 (528h23m43.500863107s) [noise=true , address=145.79.227.48:4301 , piece_id=LDACISJDANBDHKKUKGYWN4FIIDJWTGDGD3SOOE5ZUXZK2XXOLQHQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.630196715 (528h23m43.50127975s) [noise=true , address=145.79.227.52:4445 , piece_id=BLM6FGNXUD7HPU5JDRZE6YXZPKF3QTYVXJLEHMUVA6URPGLZGGWA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.630281747 (528h23m43.501364782s) [noise=true , address=145.79.227.11:4730 , piece_id=IYFN7VTLD6I4Y2UUL2Q3URQSNEGB37XEYIEAYEQRZL6VRECZZHUQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.633122186 (528h23m43.504205221s) [noise=true , address=145.79.227.4:4186 , piece_id=2VV7M2LTB3TTBMIHPY5UJVNHHLHCRAGAFBQNW4OBGSQEEES2Y2CA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.641421739 (528h23m43.512504774s) [noise=true , address=145.79.227.5:4223 , piece_id=2T2YRLBMQKHE6KKKNKAFBHO5EZDZDP6S2CWB3H4IAXVJXMXF5PZQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.646197363 (528h23m43.517280408s) [noise=true , address=145.79.227.51:4409 , piece_id=LNMYX6DXKV6OYNBGK4D4AMIPPXXUFADUUKYH4N3Y6ZNYWV4PZVWA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.667657547 (528h23m43.538740582s) [noise=true , address=145.79.227.49:4339 , piece_id=5PAETDZASQLO47SSXBHLLTFYXORQS27WKIIAMHRVWISO6FHRORXA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.824500013 (528h23m43.695583048s) [noise=true , address=77.90.31.18:4808 , piece_id=7L7U3VR63WOGY4LDT5E3CYNTGADMRBNXX4RNYHSZ77MQSGR34AMQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.913399012 (528h23m43.784482067s) [noise=true , address=77.90.31.16:4735 , piece_id=P7LT6TGLWHV7RUR3CL2532TKOB2XZHGHN26V7CMULNXA2FDGHOGA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.984027901 (528h23m43.855110937s) [noise=true , address=45.91.116.4:4939 , piece_id=PRZXTLWPCI36PFH3J5FPPHZIJPJEHUZIKGZLVKQV5Y7DVXQONSPA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:44.012309846 (528h23m43.883392881s) [noise=true , address=45.91.116.11:4118 , piece_id=TR7P4KFGCQTOLTJ45PGKOAQZ5I45XJ5NPYHGCLEFYFVET5TGDQVA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:44.012331005 (528h23m43.88341404s) [seg_pos={0 0}, upload=463448]: Segment reached optimal threshold of 13 pieces. Dec 27 15:52:44.012552147 (528h23m43.883635192s) [noise=true , address=45.91.116.5:4298 , piece_id=OXCFVA5L3YKUJOPSSQJ3HAIZ77KDKFLI4B2KLE6BFMTYLKSFU54Q, seg_pos={0 0}, upload=463448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 15:52:44.012559651 (528h23m43.883642686s) [noise=true , address=45.91.116.5:4298 , piece_id=OXCFVA5L3YKUJOPSSQJ3HAIZ77KDKFLI4B2KLE6BFMTYLKSFU54Q, seg_pos={0 0}, upload=463448]: Long tail cancellation due to optimal threshold Dec 27 15:52:44.012718022 (528h23m43.883801077s) [noise=true , address=45.91.116.13:4154 , piece_id=3AEGYH2XRZ4YJ2I5A7WGR4VPUOQTSQAU4CW4P6ERTO5UMTP6NDTQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 15:52:44.012737700 (528h23m43.883820745s) [noise=true , address=45.91.116.13:4154 , piece_id=3AEGYH2XRZ4YJ2I5A7WGR4VPUOQTSQAU4CW4P6ERTO5UMTP6NDTQ, seg_pos={0 0}, upload=463448]: Long tail cancellation due to optimal threshold Dec 27 15:52:44.012773749 (528h23m43.883856784s) [noise=true , address=77.90.31.11:4556 , piece_id=QSOG73Y6IGJ334UYSHF4HOWNXIQD635C7NJ66UPDS4CSPUQAFTDA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: read tcp 188.166.225.170:60458->77.90.31.11:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:52:44.012781493 (528h23m43.883864538s) [noise=true , address=77.90.31.11:4556 , piece_id=QSOG73Y6IGJ334UYSHF4HOWNXIQD635C7NJ66UPDS4CSPUQAFTDA, seg_pos={0 0}, upload=463448]: Long tail cancellation due to optimal threshold Dec 27 15:52:44.012847208 (528h23m43.883930254s) [seg_pos={0 0}, upload=463448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:52:44.012976224 (528h23m43.88405926s) [upload=463448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:52:44.012983658 (528h23m43.884066703s) [upload=463448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:52:44.338288516 (528h23m44.209371551s) [upload=463448]: Done uploading segments... Dec 27 15:52:44.338312862 (528h23m44.209395897s) [upload=463448]: Upload finished. err: <nil> Duration: 5.62510274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X.....XX......X................XX........] duration: 49.146022284s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:53:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 49.146037733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX.....] duration: 5.000796713s trace: https://jaeger.dev.storj.io/trace/534a269c5f8f3f97 FAIL (2025-12-27 15:53:12 UTC) orbiter warning: context deadline exceeded Duration: 5.00083191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.X..X.X.....X...X......X.X..X..XXX...........X.......X..X] duration: 30.004647334s trace: https://jaeger.dev.storj.io/trace/7cf60a64b935d292 FAIL (2025-12-27 15:53:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TR7P4KFGCQTOLTJ45PGKOAQZ5I45XJ5NPYHGCLEFYFVET5TGDQVA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 7L7U3VR63WOGY4LDT5E3CYNTGADMRBNXX4RNYHSZ77MQSGR34AMQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: P7LT6TGLWHV7RUR3CL2532TKOB2XZHGHN26V7CMULNXA2FDGHOGA) context deadline exceeded ======LOGS====== Dec 27 15:52:41.366331860 (528h23m41.237414895s) [upload=463448]: Starting upload Dec 27 15:52:41.366391583 (528h23m41.237474618s) [upload=463448]: Done starting upload Dec 27 15:52:41.366498497 (528h23m41.237581542s) [upload=463448]: Uploading segments... Dec 27 15:52:41.366679592 (528h23m41.237762627s) [upload=463448]: Got next segment. Inline: false Dec 27 15:52:41.366754415 (528h23m41.23783746s) [upload=463448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:52:41.366760496 (528h23m41.237843541s) [upload=463448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:52:41.617621119 (528h23m41.488704164s) [seg_pos={0 0}, upload=463448]: Begin upload segment... Dec 27 15:52:41.617848743 (528h23m41.488931788s) [seg_pos={0 0}, upload=463448]: Done begin upload segment. Dec 27 15:52:41.617907554 (528h23m41.488990599s) [noise=true , address=145.79.227.11:4730 , piece_id=IYFN7VTLD6I4Y2UUL2Q3URQSNEGB37XEYIEAYEQRZL6VRECZZHUQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.617926300 (528h23m41.489009345s) [noise=true , address=145.79.227.51:4409 , piece_id=LNMYX6DXKV6OYNBGK4D4AMIPPXXUFADUUKYH4N3Y6ZNYWV4PZVWA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.618600894 (528h23m41.489683939s) [noise=true , address=145.79.227.49:4339 , piece_id=5PAETDZASQLO47SSXBHLLTFYXORQS27WKIIAMHRVWISO6FHRORXA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.618709451 (528h23m41.489792496s) [noise=true , address=45.91.116.4:4939 , piece_id=PRZXTLWPCI36PFH3J5FPPHZIJPJEHUZIKGZLVKQV5Y7DVXQONSPA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.619061641 (528h23m41.490144676s) [noise=true , address=45.91.116.11:4118 , piece_id=TR7P4KFGCQTOLTJ45PGKOAQZ5I45XJ5NPYHGCLEFYFVET5TGDQVA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.619263416 (528h23m41.490346451s) [noise=true , address=145.79.227.5:4223 , piece_id=2T2YRLBMQKHE6KKKNKAFBHO5EZDZDP6S2CWB3H4IAXVJXMXF5PZQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.619616217 (528h23m41.490699252s) [noise=true , address=77.90.31.11:4556 , piece_id=QSOG73Y6IGJ334UYSHF4HOWNXIQD635C7NJ66UPDS4CSPUQAFTDA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.619758299 (528h23m41.490841344s) [noise=true , address=145.79.227.52:4445 , piece_id=BLM6FGNXUD7HPU5JDRZE6YXZPKF3QTYVXJLEHMUVA6URPGLZGGWA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.620060964 (528h23m41.491143999s) [noise=true , address=77.90.31.18:4808 , piece_id=7L7U3VR63WOGY4LDT5E3CYNTGADMRBNXX4RNYHSZ77MQSGR34AMQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.620274010 (528h23m41.491357045s) [noise=true , address=77.90.31.16:4735 , piece_id=P7LT6TGLWHV7RUR3CL2532TKOB2XZHGHN26V7CMULNXA2FDGHOGA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.620561077 (528h23m41.491644122s) [noise=true , address=45.91.116.13:4154 , piece_id=3AEGYH2XRZ4YJ2I5A7WGR4VPUOQTSQAU4CW4P6ERTO5UMTP6NDTQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.620792708 (528h23m41.491875753s) [noise=true , address=145.79.227.42:4121 , piece_id=6PM7JIBT6G5C43DZK5VOOQCQX2H4P3HNEWCYRTH43D5P3ITSASKQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.621283582 (528h23m41.492366617s) [noise=true , address=45.91.116.5:4298 , piece_id=OXCFVA5L3YKUJOPSSQJ3HAIZ77KDKFLI4B2KLE6BFMTYLKSFU54Q, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.621843879 (528h23m41.492926924s) [noise=true , address=145.79.227.4:4186 , piece_id=2VV7M2LTB3TTBMIHPY5UJVNHHLHCRAGAFBQNW4OBGSQEEES2Y2CA, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.622356886 (528h23m41.493439931s) [noise=true , address=145.79.227.48:4301 , piece_id=LDACISJDANBDHKKUKGYWN4FIIDJWTGDGD3SOOE5ZUXZK2XXOLQHQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:41.623061969 (528h23m41.494145014s) [noise=true , address=145.79.227.10:4402 , piece_id=EPTGZDYB6URVLRJAIUXELVA6JYCXNDA5AEFJ5WWVP77UBNPDLHPQ, seg_pos={0 0}, upload=463448]: Uploading piece... Dec 27 15:52:43.242171828 (528h23m43.113254863s) [upload=463448]: Next returned nil segment Dec 27 15:52:43.242184562 (528h23m43.113267607s) [upload=463448]: Waiting for error group managing segments... Dec 27 15:52:43.627203516 (528h23m43.498286561s) [noise=true , address=145.79.227.10:4402 , piece_id=EPTGZDYB6URVLRJAIUXELVA6JYCXNDA5AEFJ5WWVP77UBNPDLHPQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.628848718 (528h23m43.499931753s) [noise=true , address=145.79.227.42:4121 , piece_id=6PM7JIBT6G5C43DZK5VOOQCQX2H4P3HNEWCYRTH43D5P3ITSASKQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.629780062 (528h23m43.500863107s) [noise=true , address=145.79.227.48:4301 , piece_id=LDACISJDANBDHKKUKGYWN4FIIDJWTGDGD3SOOE5ZUXZK2XXOLQHQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.630196715 (528h23m43.50127975s) [noise=true , address=145.79.227.52:4445 , piece_id=BLM6FGNXUD7HPU5JDRZE6YXZPKF3QTYVXJLEHMUVA6URPGLZGGWA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.630281747 (528h23m43.501364782s) [noise=true , address=145.79.227.11:4730 , piece_id=IYFN7VTLD6I4Y2UUL2Q3URQSNEGB37XEYIEAYEQRZL6VRECZZHUQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.633122186 (528h23m43.504205221s) [noise=true , address=145.79.227.4:4186 , piece_id=2VV7M2LTB3TTBMIHPY5UJVNHHLHCRAGAFBQNW4OBGSQEEES2Y2CA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.641421739 (528h23m43.512504774s) [noise=true , address=145.79.227.5:4223 , piece_id=2T2YRLBMQKHE6KKKNKAFBHO5EZDZDP6S2CWB3H4IAXVJXMXF5PZQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.646197363 (528h23m43.517280408s) [noise=true , address=145.79.227.51:4409 , piece_id=LNMYX6DXKV6OYNBGK4D4AMIPPXXUFADUUKYH4N3Y6ZNYWV4PZVWA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.667657547 (528h23m43.538740582s) [noise=true , address=145.79.227.49:4339 , piece_id=5PAETDZASQLO47SSXBHLLTFYXORQS27WKIIAMHRVWISO6FHRORXA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.824500013 (528h23m43.695583048s) [noise=true , address=77.90.31.18:4808 , piece_id=7L7U3VR63WOGY4LDT5E3CYNTGADMRBNXX4RNYHSZ77MQSGR34AMQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.913399012 (528h23m43.784482067s) [noise=true , address=77.90.31.16:4735 , piece_id=P7LT6TGLWHV7RUR3CL2532TKOB2XZHGHN26V7CMULNXA2FDGHOGA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:43.984027901 (528h23m43.855110937s) [noise=true , address=45.91.116.4:4939 , piece_id=PRZXTLWPCI36PFH3J5FPPHZIJPJEHUZIKGZLVKQV5Y7DVXQONSPA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:44.012309846 (528h23m43.883392881s) [noise=true , address=45.91.116.11:4118 , piece_id=TR7P4KFGCQTOLTJ45PGKOAQZ5I45XJ5NPYHGCLEFYFVET5TGDQVA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: <nil> Dec 27 15:52:44.012331005 (528h23m43.88341404s) [seg_pos={0 0}, upload=463448]: Segment reached optimal threshold of 13 pieces. Dec 27 15:52:44.012552147 (528h23m43.883635192s) [noise=true , address=45.91.116.5:4298 , piece_id=OXCFVA5L3YKUJOPSSQJ3HAIZ77KDKFLI4B2KLE6BFMTYLKSFU54Q, seg_pos={0 0}, upload=463448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 15:52:44.012559651 (528h23m43.883642686s) [noise=true , address=45.91.116.5:4298 , piece_id=OXCFVA5L3YKUJOPSSQJ3HAIZ77KDKFLI4B2KLE6BFMTYLKSFU54Q, seg_pos={0 0}, upload=463448]: Long tail cancellation due to optimal threshold Dec 27 15:52:44.012718022 (528h23m43.883801077s) [noise=true , address=45.91.116.13:4154 , piece_id=3AEGYH2XRZ4YJ2I5A7WGR4VPUOQTSQAU4CW4P6ERTO5UMTP6NDTQ, seg_pos={0 0}, upload=463448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 15:52:44.012737700 (528h23m43.883820745s) [noise=true , address=45.91.116.13:4154 , piece_id=3AEGYH2XRZ4YJ2I5A7WGR4VPUOQTSQAU4CW4P6ERTO5UMTP6NDTQ, seg_pos={0 0}, upload=463448]: Long tail cancellation due to optimal threshold Dec 27 15:52:44.012773749 (528h23m43.883856784s) [noise=true , address=77.90.31.11:4556 , piece_id=QSOG73Y6IGJ334UYSHF4HOWNXIQD635C7NJ66UPDS4CSPUQAFTDA, seg_pos={0 0}, upload=463448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: read tcp 188.166.225.170:60458->77.90.31.11:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:52:44.012781493 (528h23m43.883864538s) [noise=true , address=77.90.31.11:4556 , piece_id=QSOG73Y6IGJ334UYSHF4HOWNXIQD635C7NJ66UPDS4CSPUQAFTDA, seg_pos={0 0}, upload=463448]: Long tail cancellation due to optimal threshold Dec 27 15:52:44.012847208 (528h23m43.883930254s) [seg_pos={0 0}, upload=463448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:52:44.012976224 (528h23m43.88405926s) [upload=463448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:52:44.012983658 (528h23m43.884066703s) [upload=463448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:52:44.338288516 (528h23m44.209371551s) [upload=463448]: Done uploading segments... Dec 27 15:52:44.338312862 (528h23m44.209395897s) [upload=463448]: Upload finished. err: <nil> Duration: 30.004671961s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...................X.....XX......X................XX.......] duration: 40.175887714s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:52:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 40.17590606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X.X..X.X.....X...X......X.X..X..XXX...........X.......X..] duration: 6.071902444s trace: https://jaeger.dev.storj.io/trace/697bd9b30eaf7c57 FAIL (2025-12-27 15:52:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: YEFN2DHALZKYOFOEZC4BL3DEESMAGFHDVOBV5ON4NST7VYGZHCAQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: WZPX5FGHLNR666PZHCAET6H3AQV7A76BSOTR6D4F6XYZKDCGAWWA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 5ORLFJNDRJJZ2ENB22Q5KS653NTF34BXOCEIXYAJEPMHFQITUVFQ) context deadline exceeded ======LOGS====== ec 27 15:50:49.409790250 (528h21m49.280873285s) [upload=463419]: Starting upload Dec 27 15:50:49.409950195 (528h21m49.28103323s) [upload=463419]: Done starting upload Dec 27 15:50:49.410089400 (528h21m49.281172466s) [upload=463419]: Uploading segments... Dec 27 15:50:49.410312124 (528h21m49.281395199s) [upload=463419]: Got next segment. Inline: false Dec 27 15:50:49.410369734 (528h21m49.281452779s) [upload=463419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:50:49.410377128 (528h21m49.281460193s) [upload=463419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:50:49.662407598 (528h21m49.533490633s) [seg_pos={0 0}, upload=463419]: Begin upload segment... Dec 27 15:50:49.662471409 (528h21m49.533554454s) [seg_pos={0 0}, upload=463419]: Done begin upload segment. Dec 27 15:50:49.662526074 (528h21m49.533609109s) [noise=true , address=45.91.116.11:4119 , piece_id=Q5BBL7CQDXCM6MKVRSEV7JJNAYCABCTPGZJFL36ULEQ4OPKPVSNA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.662561892 (528h21m49.533644927s) [noise=true , address=145.79.227.26:4655 , piece_id=CNC7D54FJ4ICC53BGFDHCX4RZJF2VAOHTDRXX5L4LV7RMFKLFO4A, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.663330235 (528h21m49.53441327s) [noise=true , address=145.79.227.5:4221 , piece_id=V6DFQCN2ZY4OPMVLP43CJLJFCHFZI5QHVABFIREDRQBNNO6HC4CA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.663787015 (528h21m49.53487005s) [noise=true , address=145.79.227.50:4374 , piece_id=BMQLHA2ECQ4LAEFLCQDBT7MG5766XQUIEDNGXNHQGEQ4FTTK65FQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.664098628 (528h21m49.535181663s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPX5FGHLNR666PZHCAET6H3AQV7A76BSOTR6D4F6XYZKDCGAWWA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.664386566 (528h21m49.535469601s) [noise=true , address=45.91.116.4:4937 , piece_id=TCTLWQXUDTA6CIATYNHJJUR553KUBFY4WJYTGKXMTJD2UMN5DZUA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.664678643 (528h21m49.535761688s) [noise=true , address=145.79.227.21:4474 , piece_id=UIJF3ZK722NDGJZCZFU3MEJ3UZJFJHC2TJUQFU7YRCCDPQG7VSLQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.664952204 (528h21m49.536035249s) [noise=true , address=45.91.116.5:4297 , piece_id=5ORLFJNDRJJZ2ENB22Q5KS653NTF34BXOCEIXYAJEPMHFQITUVFQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.665237347 (528h21m49.536320382s) [noise=true , address=145.79.227.4:4185 , piece_id=XXKU56DYUATEIHWPDIPBMEDQ6ZUG2BDRNAZZXUDSW5WHYSZAUILQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.666300300 (528h21m49.537383345s) [noise=true , address=145.79.227.51:4409 , piece_id=6O4MM3EJ7XRFPL54353IR6C5BD6BM5TJ5WWDNOHAXDJB5TFBXQ3A, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.666634507 (528h21m49.537717542s) [noise=true , address=145.79.227.43:4157 , piece_id=CUYIMNZIVLU4XRMBO3YUHE6BJAIXEQ2MLQGTBJ4MDVPATJS7A2PQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.666980145 (528h21m49.53806318s) [noise=true , address=145.79.227.11:4732 , piece_id=KE47KCSKY34G4LTC6EG6Z56JDRI7XH4EDPIHOXDLCLJI4P3HTXTA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.667980350 (528h21m49.539063385s) [noise=true , address=45.91.116.6:4976 , piece_id=6CRDE2Q6WICW4GLEBZRVE4DOU6HRWKI64K7FUTL6N7NCHJRJFSFA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.669873805 (528h21m49.54095694s) [noise=true , address=45.91.116.12:4226 , piece_id=YEFN2DHALZKYOFOEZC4BL3DEESMAGFHDVOBV5ON4NST7VYGZHCAQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.683807667 (528h21m49.554890712s) [noise=true , address=145.79.227.42:4121 , piece_id=WL4Z5NMADMWDWEK6PJUTT3GTCBR3Y3TWVJTSBI2VAV3WXDKVVCCQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.686389233 (528h21m49.557472358s) [noise=true , address=145.79.227.9:4365 , piece_id=LIU5RRYWI4CRFG5DV5DS3AQLAJICNG7EK5LKEL3SU3GMC4YJ6IPQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:51.391946062 (528h21m51.263029117s) [upload=463419]: Next returned nil segment Dec 27 15:50:51.391991739 (528h21m51.263074784s) [upload=463419]: Waiting for error group managing segments... Dec 27 15:50:51.674658486 (528h21m51.545741531s) [noise=true , address=145.79.227.51:4409 , piece_id=6O4MM3EJ7XRFPL54353IR6C5BD6BM5TJ5WWDNOHAXDJB5TFBXQ3A, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.676883553 (528h21m51.547966588s) [noise=true , address=145.79.227.4:4185 , piece_id=XXKU56DYUATEIHWPDIPBMEDQ6ZUG2BDRNAZZXUDSW5WHYSZAUILQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.678845479 (528h21m51.549928524s) [noise=true , address=145.79.227.21:4474 , piece_id=UIJF3ZK722NDGJZCZFU3MEJ3UZJFJHC2TJUQFU7YRCCDPQG7VSLQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.679344069 (528h21m51.550427134s) [noise=true , address=145.79.227.50:4374 , piece_id=BMQLHA2ECQ4LAEFLCQDBT7MG5766XQUIEDNGXNHQGEQ4FTTK65FQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.682425876 (528h21m51.553508922s) [noise=true , address=145.79.227.5:4221 , piece_id=V6DFQCN2ZY4OPMVLP43CJLJFCHFZI5QHVABFIREDRQBNNO6HC4CA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.692469141 (528h21m51.563552186s) [noise=true , address=145.79.227.9:4365 , piece_id=LIU5RRYWI4CRFG5DV5DS3AQLAJICNG7EK5LKEL3SU3GMC4YJ6IPQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.695298128 (528h21m51.566381163s) [noise=true , address=145.79.227.42:4121 , piece_id=WL4Z5NMADMWDWEK6PJUTT3GTCBR3Y3TWVJTSBI2VAV3WXDKVVCCQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.695720572 (528h21m51.566803617s) [noise=true , address=145.79.227.26:4655 , piece_id=CNC7D54FJ4ICC53BGFDHCX4RZJF2VAOHTDRXX5L4LV7RMFKLFO4A, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.696120334 (528h21m51.567203369s) [noise=true , address=145.79.227.11:4732 , piece_id=KE47KCSKY34G4LTC6EG6Z56JDRI7XH4EDPIHOXDLCLJI4P3HTXTA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.877734278 (528h21m51.748817313s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPX5FGHLNR666PZHCAET6H3AQV7A76BSOTR6D4F6XYZKDCGAWWA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.911997921 (528h21m51.783080956s) [noise=true , address=45.91.116.12:4226 , piece_id=YEFN2DHALZKYOFOEZC4BL3DEESMAGFHDVOBV5ON4NST7VYGZHCAQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.931104163 (528h21m51.802187208s) [noise=true , address=45.91.116.5:4297 , piece_id=5ORLFJNDRJJZ2ENB22Q5KS653NTF34BXOCEIXYAJEPMHFQITUVFQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.984569641 (528h21m51.855652706s) [noise=true , address=45.91.116.6:4976 , piece_id=6CRDE2Q6WICW4GLEBZRVE4DOU6HRWKI64K7FUTL6N7NCHJRJFSFA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.984594619 (528h21m51.855677664s) [seg_pos={0 0}, upload=463419]: Segment reached optimal threshold of 13 pieces. Dec 27 15:50:51.984819688 (528h21m51.855902723s) [noise=true , address=145.79.227.43:4157 , piece_id=CUYIMNZIVLU4XRMBO3YUHE6BJAIXEQ2MLQGTBJ4MDVPATJS7A2PQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 27 15:50:51.984827693 (528h21m51.855910738s) [noise=true , address=145.79.227.43:4157 , piece_id=CUYIMNZIVLU4XRMBO3YUHE6BJAIXEQ2MLQGTBJ4MDVPATJS7A2PQ, seg_pos={0 0}, upload=463419]: Long tail cancellation due to optimal threshold Dec 27 15:50:51.984864022 (528h21m51.855947077s) [noise=true , address=45.91.116.11:4119 , piece_id=Q5BBL7CQDXCM6MKVRSEV7JJNAYCABCTPGZJFL36ULEQ4OPKPVSNA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 15:50:51.984874241 (528h21m51.855957286s) [noise=true , address=45.91.116.11:4119 , piece_id=Q5BBL7CQDXCM6MKVRSEV7JJNAYCABCTPGZJFL36ULEQ4OPKPVSNA, seg_pos={0 0}, upload=463419]: Long tail cancellation due to optimal threshold Dec 27 15:50:51.985097086 (528h21m51.856180121s) [noise=true , address=45.91.116.4:4937 , piece_id=TCTLWQXUDTA6CIATYNHJJUR553KUBFY4WJYTGKXMTJD2UMN5DZUA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40970->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40970->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40970->45.91.116.4:4937: use of closed network connection Dec 27 15:50:51.985107305 (528h21m51.85619034s) [noise=true , address=45.91.116.4:4937 , piece_id=TCTLWQXUDTA6CIATYNHJJUR553KUBFY4WJYTGKXMTJD2UMN5DZUA, seg_pos={0 0}, upload=463419]: Long tail cancellation due to optimal threshold Dec 27 15:50:51.985129316 (528h21m51.856212361s) [seg_pos={0 0}, upload=463419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:50:51.985215721 (528h21m51.856298757s) [upload=463419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:50:51.985223456 (528h21m51.856306491s) [upload=463419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:50:52.274261756 (528h21m52.145344801s) [upload=463419]: Done uploading segments... Dec 27 15:50:52.274283066 (528h21m52.145366111s) [upload=463419]: Upload finished. err: <nil> Duration: 6.071928035s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX....] duration: 5.000758319s trace: https://jaeger.dev.storj.io/trace/534a269c5f8f3f97 FAIL (2025-12-27 15:52:01 UTC) orbiter warning: context deadline exceeded Duration: 5.000787264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...................X.....XX......X................XX......] duration: 49.283868616s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 15:51:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 49.283987052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX....XXXX..XX..XX...] duration: 5.000825037s trace: https://jaeger.dev.storj.io/trace/534a269c5f8f3f97 FAIL (2025-12-27 15:51:39 UTC) orbiter warning: context deadline exceeded Duration: 5.000857849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX..X.X..X.X.....X...X......X.X..X..XXX...........X.......X.] duration: 5.647706712s trace: https://jaeger.dev.storj.io/trace/697bd9b30eaf7c57 FAIL (2025-12-27 15:51:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: YEFN2DHALZKYOFOEZC4BL3DEESMAGFHDVOBV5ON4NST7VYGZHCAQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: WZPX5FGHLNR666PZHCAET6H3AQV7A76BSOTR6D4F6XYZKDCGAWWA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 5ORLFJNDRJJZ2ENB22Q5KS653NTF34BXOCEIXYAJEPMHFQITUVFQ) context deadline exceeded ======LOGS====== ec 27 15:50:49.409790250 (528h21m49.280873285s) [upload=463419]: Starting upload Dec 27 15:50:49.409950195 (528h21m49.28103323s) [upload=463419]: Done starting upload Dec 27 15:50:49.410089400 (528h21m49.281172466s) [upload=463419]: Uploading segments... Dec 27 15:50:49.410312124 (528h21m49.281395199s) [upload=463419]: Got next segment. Inline: false Dec 27 15:50:49.410369734 (528h21m49.281452779s) [upload=463419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:50:49.410377128 (528h21m49.281460193s) [upload=463419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:50:49.662407598 (528h21m49.533490633s) [seg_pos={0 0}, upload=463419]: Begin upload segment... Dec 27 15:50:49.662471409 (528h21m49.533554454s) [seg_pos={0 0}, upload=463419]: Done begin upload segment. Dec 27 15:50:49.662526074 (528h21m49.533609109s) [noise=true , address=45.91.116.11:4119 , piece_id=Q5BBL7CQDXCM6MKVRSEV7JJNAYCABCTPGZJFL36ULEQ4OPKPVSNA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.662561892 (528h21m49.533644927s) [noise=true , address=145.79.227.26:4655 , piece_id=CNC7D54FJ4ICC53BGFDHCX4RZJF2VAOHTDRXX5L4LV7RMFKLFO4A, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.663330235 (528h21m49.53441327s) [noise=true , address=145.79.227.5:4221 , piece_id=V6DFQCN2ZY4OPMVLP43CJLJFCHFZI5QHVABFIREDRQBNNO6HC4CA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.663787015 (528h21m49.53487005s) [noise=true , address=145.79.227.50:4374 , piece_id=BMQLHA2ECQ4LAEFLCQDBT7MG5766XQUIEDNGXNHQGEQ4FTTK65FQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.664098628 (528h21m49.535181663s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPX5FGHLNR666PZHCAET6H3AQV7A76BSOTR6D4F6XYZKDCGAWWA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.664386566 (528h21m49.535469601s) [noise=true , address=45.91.116.4:4937 , piece_id=TCTLWQXUDTA6CIATYNHJJUR553KUBFY4WJYTGKXMTJD2UMN5DZUA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.664678643 (528h21m49.535761688s) [noise=true , address=145.79.227.21:4474 , piece_id=UIJF3ZK722NDGJZCZFU3MEJ3UZJFJHC2TJUQFU7YRCCDPQG7VSLQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.664952204 (528h21m49.536035249s) [noise=true , address=45.91.116.5:4297 , piece_id=5ORLFJNDRJJZ2ENB22Q5KS653NTF34BXOCEIXYAJEPMHFQITUVFQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.665237347 (528h21m49.536320382s) [noise=true , address=145.79.227.4:4185 , piece_id=XXKU56DYUATEIHWPDIPBMEDQ6ZUG2BDRNAZZXUDSW5WHYSZAUILQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.666300300 (528h21m49.537383345s) [noise=true , address=145.79.227.51:4409 , piece_id=6O4MM3EJ7XRFPL54353IR6C5BD6BM5TJ5WWDNOHAXDJB5TFBXQ3A, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.666634507 (528h21m49.537717542s) [noise=true , address=145.79.227.43:4157 , piece_id=CUYIMNZIVLU4XRMBO3YUHE6BJAIXEQ2MLQGTBJ4MDVPATJS7A2PQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.666980145 (528h21m49.53806318s) [noise=true , address=145.79.227.11:4732 , piece_id=KE47KCSKY34G4LTC6EG6Z56JDRI7XH4EDPIHOXDLCLJI4P3HTXTA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.667980350 (528h21m49.539063385s) [noise=true , address=45.91.116.6:4976 , piece_id=6CRDE2Q6WICW4GLEBZRVE4DOU6HRWKI64K7FUTL6N7NCHJRJFSFA, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.669873805 (528h21m49.54095694s) [noise=true , address=45.91.116.12:4226 , piece_id=YEFN2DHALZKYOFOEZC4BL3DEESMAGFHDVOBV5ON4NST7VYGZHCAQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.683807667 (528h21m49.554890712s) [noise=true , address=145.79.227.42:4121 , piece_id=WL4Z5NMADMWDWEK6PJUTT3GTCBR3Y3TWVJTSBI2VAV3WXDKVVCCQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:49.686389233 (528h21m49.557472358s) [noise=true , address=145.79.227.9:4365 , piece_id=LIU5RRYWI4CRFG5DV5DS3AQLAJICNG7EK5LKEL3SU3GMC4YJ6IPQ, seg_pos={0 0}, upload=463419]: Uploading piece... Dec 27 15:50:51.391946062 (528h21m51.263029117s) [upload=463419]: Next returned nil segment Dec 27 15:50:51.391991739 (528h21m51.263074784s) [upload=463419]: Waiting for error group managing segments... Dec 27 15:50:51.674658486 (528h21m51.545741531s) [noise=true , address=145.79.227.51:4409 , piece_id=6O4MM3EJ7XRFPL54353IR6C5BD6BM5TJ5WWDNOHAXDJB5TFBXQ3A, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.676883553 (528h21m51.547966588s) [noise=true , address=145.79.227.4:4185 , piece_id=XXKU56DYUATEIHWPDIPBMEDQ6ZUG2BDRNAZZXUDSW5WHYSZAUILQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.678845479 (528h21m51.549928524s) [noise=true , address=145.79.227.21:4474 , piece_id=UIJF3ZK722NDGJZCZFU3MEJ3UZJFJHC2TJUQFU7YRCCDPQG7VSLQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.679344069 (528h21m51.550427134s) [noise=true , address=145.79.227.50:4374 , piece_id=BMQLHA2ECQ4LAEFLCQDBT7MG5766XQUIEDNGXNHQGEQ4FTTK65FQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.682425876 (528h21m51.553508922s) [noise=true , address=145.79.227.5:4221 , piece_id=V6DFQCN2ZY4OPMVLP43CJLJFCHFZI5QHVABFIREDRQBNNO6HC4CA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.692469141 (528h21m51.563552186s) [noise=true , address=145.79.227.9:4365 , piece_id=LIU5RRYWI4CRFG5DV5DS3AQLAJICNG7EK5LKEL3SU3GMC4YJ6IPQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.695298128 (528h21m51.566381163s) [noise=true , address=145.79.227.42:4121 , piece_id=WL4Z5NMADMWDWEK6PJUTT3GTCBR3Y3TWVJTSBI2VAV3WXDKVVCCQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.695720572 (528h21m51.566803617s) [noise=true , address=145.79.227.26:4655 , piece_id=CNC7D54FJ4ICC53BGFDHCX4RZJF2VAOHTDRXX5L4LV7RMFKLFO4A, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.696120334 (528h21m51.567203369s) [noise=true , address=145.79.227.11:4732 , piece_id=KE47KCSKY34G4LTC6EG6Z56JDRI7XH4EDPIHOXDLCLJI4P3HTXTA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.877734278 (528h21m51.748817313s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPX5FGHLNR666PZHCAET6H3AQV7A76BSOTR6D4F6XYZKDCGAWWA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.911997921 (528h21m51.783080956s) [noise=true , address=45.91.116.12:4226 , piece_id=YEFN2DHALZKYOFOEZC4BL3DEESMAGFHDVOBV5ON4NST7VYGZHCAQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.931104163 (528h21m51.802187208s) [noise=true , address=45.91.116.5:4297 , piece_id=5ORLFJNDRJJZ2ENB22Q5KS653NTF34BXOCEIXYAJEPMHFQITUVFQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.984569641 (528h21m51.855652706s) [noise=true , address=45.91.116.6:4976 , piece_id=6CRDE2Q6WICW4GLEBZRVE4DOU6HRWKI64K7FUTL6N7NCHJRJFSFA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: <nil> Dec 27 15:50:51.984594619 (528h21m51.855677664s) [seg_pos={0 0}, upload=463419]: Segment reached optimal threshold of 13 pieces. Dec 27 15:50:51.984819688 (528h21m51.855902723s) [noise=true , address=145.79.227.43:4157 , piece_id=CUYIMNZIVLU4XRMBO3YUHE6BJAIXEQ2MLQGTBJ4MDVPATJS7A2PQ, seg_pos={0 0}, upload=463419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 27 15:50:51.984827693 (528h21m51.855910738s) [noise=true , address=145.79.227.43:4157 , piece_id=CUYIMNZIVLU4XRMBO3YUHE6BJAIXEQ2MLQGTBJ4MDVPATJS7A2PQ, seg_pos={0 0}, upload=463419]: Long tail cancellation due to optimal threshold Dec 27 15:50:51.984864022 (528h21m51.855947077s) [noise=true , address=45.91.116.11:4119 , piece_id=Q5BBL7CQDXCM6MKVRSEV7JJNAYCABCTPGZJFL36ULEQ4OPKPVSNA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 15:50:51.984874241 (528h21m51.855957286s) [noise=true , address=45.91.116.11:4119 , piece_id=Q5BBL7CQDXCM6MKVRSEV7JJNAYCABCTPGZJFL36ULEQ4OPKPVSNA, seg_pos={0 0}, upload=463419]: Long tail cancellation due to optimal threshold Dec 27 15:50:51.985097086 (528h21m51.856180121s) [noise=true , address=45.91.116.4:4937 , piece_id=TCTLWQXUDTA6CIATYNHJJUR553KUBFY4WJYTGKXMTJD2UMN5DZUA, seg_pos={0 0}, upload=463419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40970->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40970->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40970->45.91.116.4:4937: use of closed network connection Dec 27 15:50:51.985107305 (528h21m51.85619034s) [noise=true , address=45.91.116.4:4937 , piece_id=TCTLWQXUDTA6CIATYNHJJUR553KUBFY4WJYTGKXMTJD2UMN5DZUA, seg_pos={0 0}, upload=463419]: Long tail cancellation due to optimal threshold Dec 27 15:50:51.985129316 (528h21m51.856212361s) [seg_pos={0 0}, upload=463419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:50:51.985215721 (528h21m51.856298757s) [upload=463419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:50:51.985223456 (528h21m51.856306491s) [upload=463419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:50:52.274261756 (528h21m52.145344801s) [upload=463419]: Done uploading segments... Dec 27 15:50:52.274283066 (528h21m52.145366111s) [upload=463419]: Upload finished. err: <nil> Duration: 5.647734806s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)