Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [..X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..X] duration: 5.001525978s trace: https://jaeger.dev.storj.io/trace/1e1367da861e4222 FAIL (2025-12-28 12:00:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001552358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 12.123962317s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:00:35 UTC) orbiter warning: context deadline exceeded Duration: 12.123982095s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....................XX....X.....XX...XXX.......X..XX....X...] duration: 5.72911295s trace: https://jaeger.dev.storj.io/trace/7b7bce03ce6b14d2 FAIL (2025-12-28 12:00:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:59:04.566286472 (548h30m4.437369517s) [upload=480950]: Starting upload Dec 28 11:59:04.566375662 (548h30m4.437458707s) [upload=480950]: Done starting upload Dec 28 11:59:04.566706552 (548h30m4.437789597s) [upload=480950]: Uploading segments... Dec 28 11:59:04.566777286 (548h30m4.437860331s) [upload=480950]: Got next segment. Inline: false Dec 28 11:59:04.566818064 (548h30m4.437901109s) [upload=480950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:59:04.566824617 (548h30m4.437907662s) [upload=480950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:59:04.812423396 (548h30m4.683506441s) [seg_pos={0 0}, upload=480950]: Begin upload segment... Dec 28 11:59:04.812462571 (548h30m4.683545606s) [seg_pos={0 0}, upload=480950]: Done begin upload segment. Dec 28 11:59:04.812487297 (548h30m4.683570332s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.812549786 (548h30m4.683632831s) [noise=true , address=77.90.31.11:4556 , piece_id=IONPRCI2HTCVD2UDXKWX2JFGOHYP75GR4Q6UWWQIQS7AN42OB34A, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813033688 (548h30m4.684116733s) [noise=true , address=145.79.227.2:4115 , piece_id=XITRDAQHGOGTXGV577JUDTKMSGNGFRDBGT5CCNUWMOTXLYH5FEEQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813451043 (548h30m4.684534088s) [noise=true , address=145.79.227.16:4910 , piece_id=FXWMHX2GDSRONV35YWIB2YR3IT2ADLAW5VHSCMYIDYS4FPXS54KQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813973558 (548h30m4.685056593s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.814497024 (548h30m4.685580059s) [noise=true , address=45.91.116.14:4189 , piece_id=6KJDBBVZ47CMUZ34NM5IIBFWMKQ5FZQNZZFOWJBQU2Y2DCKU4TZQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.814944406 (548h30m4.686027441s) [noise=true , address=145.79.227.32:4836 , piece_id=3HIFE5VAZCPK7IORT6ASYBYUHUI2GSHFTVSYW7VGZZV74VNCMY4Q, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.815390937 (548h30m4.686473982s) [noise=true , address=145.79.227.11:4731 , piece_id=PSEQSLZZGS7HXUQNATQ4SIC2N7GVI7KNIKJLAW5NKWHKQC6GGPBQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.816449362 (548h30m4.687532397s) [noise=true , address=145.79.227.12:4767 , piece_id=YRAQOD234EKCP4NCXKHJPZJ3EGB55JZCVNOVOIRER6PPDALBCXPQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.817014498 (548h30m4.688097543s) [noise=true , address=77.90.31.12:4590 , piece_id=DW4C6EFH4VABYSDDHNWSPBDGRSGQB6NYK2N2AN4LQVOTQWO53KOA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.817897820 (548h30m4.688980855s) [noise=true , address=145.79.227.17:4972 , piece_id=EPC3AD3SOB66CKUHGVRGKQHRF7BIOJLXUHGT3E52NHDBRFZC44QA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.819095080 (548h30m4.690178115s) [noise=true , address=45.91.116.4:4937 , piece_id=NFNS4RKDARVLSPWOVNZMGSSJX6GMTXXR666SYYSWARU4J7O7TYXA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.821479180 (548h30m4.692562215s) [noise=true , address=145.79.227.52:4447 , piece_id=LJGA7XILGHRZSKYW6LNA6F2TRENZNWRGVJD2VRFWM6ZHW57INHXA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.823377093 (548h30m4.694460139s) [noise=true , address=145.79.227.45:4230 , piece_id=TYFXSONYH4NWYWGKEDDU24V3YRBJ67XRA2QZRJJ5CSEALNBPEUBQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.823712342 (548h30m4.694795387s) [noise=true , address=145.79.227.38:4977 , piece_id=RXZGLTY3C3U7YQ2M5BSMGNNQD7SNX4UPHN4BERW2CETBEAXJOVZA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.825471020 (548h30m4.696554066s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:06.536203398 (548h30m6.407286434s) [upload=480950]: Next returned nil segment Dec 28 11:59:06.536216755 (548h30m6.40729979s) [upload=480950]: Waiting for error group managing segments... Dec 28 11:59:06.817621144 (548h30m6.688704179s) [noise=true , address=145.79.227.38:4977 , piece_id=RXZGLTY3C3U7YQ2M5BSMGNNQD7SNX4UPHN4BERW2CETBEAXJOVZA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824356289 (548h30m6.695439324s) [noise=true , address=145.79.227.45:4230 , piece_id=TYFXSONYH4NWYWGKEDDU24V3YRBJ67XRA2QZRJJ5CSEALNBPEUBQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824889824 (548h30m6.695972869s) [noise=true , address=145.79.227.2:4115 , piece_id=XITRDAQHGOGTXGV577JUDTKMSGNGFRDBGT5CCNUWMOTXLYH5FEEQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824958796 (548h30m6.696041831s) [noise=true , address=145.79.227.11:4731 , piece_id=PSEQSLZZGS7HXUQNATQ4SIC2N7GVI7KNIKJLAW5NKWHKQC6GGPBQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.825405146 (548h30m6.696488181s) [noise=true , address=145.79.227.17:4972 , piece_id=EPC3AD3SOB66CKUHGVRGKQHRF7BIOJLXUHGT3E52NHDBRFZC44QA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.831799772 (548h30m6.702882808s) [noise=true , address=145.79.227.32:4836 , piece_id=3HIFE5VAZCPK7IORT6ASYBYUHUI2GSHFTVSYW7VGZZV74VNCMY4Q, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.837038968 (548h30m6.708122013s) [noise=true , address=145.79.227.52:4447 , piece_id=LJGA7XILGHRZSKYW6LNA6F2TRENZNWRGVJD2VRFWM6ZHW57INHXA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.837826047 (548h30m6.708909093s) [noise=true , address=145.79.227.16:4910 , piece_id=FXWMHX2GDSRONV35YWIB2YR3IT2ADLAW5VHSCMYIDYS4FPXS54KQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.846515072 (548h30m6.717598107s) [noise=true , address=145.79.227.12:4767 , piece_id=YRAQOD234EKCP4NCXKHJPZJ3EGB55JZCVNOVOIRER6PPDALBCXPQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.090006588 (548h30m6.961089633s) [noise=true , address=45.91.116.14:4189 , piece_id=6KJDBBVZ47CMUZ34NM5IIBFWMKQ5FZQNZZFOWJBQU2Y2DCKU4TZQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.100333844 (548h30m6.971416889s) [noise=true , address=77.90.31.12:4590 , piece_id=DW4C6EFH4VABYSDDHNWSPBDGRSGQB6NYK2N2AN4LQVOTQWO53KOA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.110463131 (548h30m6.981546166s) [noise=true , address=45.91.116.4:4937 , piece_id=NFNS4RKDARVLSPWOVNZMGSSJX6GMTXXR666SYYSWARU4J7O7TYXA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.135220584 (548h30m7.006303629s) [noise=true , address=77.90.31.11:4556 , piece_id=IONPRCI2HTCVD2UDXKWX2JFGOHYP75GR4Q6UWWQIQS7AN42OB34A, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.135575850 (548h30m7.006658895s) [seg_pos={0 0}, upload=480950]: Segment reached optimal threshold of 13 pieces. Dec 28 11:59:07.135851806 (548h30m7.006934851s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.135863117 (548h30m7.006946162s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.135890990 (548h30m7.006974025s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.135908623 (548h30m7.006991658s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.136129715 (548h30m7.00721275s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.136136227 (548h30m7.007219262s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.136155663 (548h30m7.007238699s) [seg_pos={0 0}, upload=480950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:59:07.136238061 (548h30m7.007321096s) [upload=480950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:59:07.136245014 (548h30m7.007328049s) [upload=480950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:59:07.420618777 (548h30m7.291701842s) [upload=480950]: Done uploading segments... Dec 28 11:59:07.420648463 (548h30m7.291731508s) [upload=480950]: Upload finished. err: <nil> Duration: 5.72913459s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...........X..X.....................................] duration: 39.242236067s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 12:00:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 39.242261445s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 16.399688836s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:00:23 UTC) orbiter warning: context deadline exceeded Duration: 16.399708785s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [...X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..] duration: 5.00132284s trace: https://jaeger.dev.storj.io/trace/1e623ef2e11a35bf FAIL (2025-12-28 12:00:07 UTC) orbiter warning: context deadline exceeded Duration: 5.001379098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....................XX....X.....XX...XXX.......X..XX....X..] duration: 5.711197226s trace: https://jaeger.dev.storj.io/trace/7b7bce03ce6b14d2 FAIL (2025-12-28 11:59:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:59:04.566286472 (548h30m4.437369517s) [upload=480950]: Starting upload Dec 28 11:59:04.566375662 (548h30m4.437458707s) [upload=480950]: Done starting upload Dec 28 11:59:04.566706552 (548h30m4.437789597s) [upload=480950]: Uploading segments... Dec 28 11:59:04.566777286 (548h30m4.437860331s) [upload=480950]: Got next segment. Inline: false Dec 28 11:59:04.566818064 (548h30m4.437901109s) [upload=480950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:59:04.566824617 (548h30m4.437907662s) [upload=480950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:59:04.812423396 (548h30m4.683506441s) [seg_pos={0 0}, upload=480950]: Begin upload segment... Dec 28 11:59:04.812462571 (548h30m4.683545606s) [seg_pos={0 0}, upload=480950]: Done begin upload segment. Dec 28 11:59:04.812487297 (548h30m4.683570332s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.812549786 (548h30m4.683632831s) [noise=true , address=77.90.31.11:4556 , piece_id=IONPRCI2HTCVD2UDXKWX2JFGOHYP75GR4Q6UWWQIQS7AN42OB34A, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813033688 (548h30m4.684116733s) [noise=true , address=145.79.227.2:4115 , piece_id=XITRDAQHGOGTXGV577JUDTKMSGNGFRDBGT5CCNUWMOTXLYH5FEEQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813451043 (548h30m4.684534088s) [noise=true , address=145.79.227.16:4910 , piece_id=FXWMHX2GDSRONV35YWIB2YR3IT2ADLAW5VHSCMYIDYS4FPXS54KQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813973558 (548h30m4.685056593s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.814497024 (548h30m4.685580059s) [noise=true , address=45.91.116.14:4189 , piece_id=6KJDBBVZ47CMUZ34NM5IIBFWMKQ5FZQNZZFOWJBQU2Y2DCKU4TZQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.814944406 (548h30m4.686027441s) [noise=true , address=145.79.227.32:4836 , piece_id=3HIFE5VAZCPK7IORT6ASYBYUHUI2GSHFTVSYW7VGZZV74VNCMY4Q, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.815390937 (548h30m4.686473982s) [noise=true , address=145.79.227.11:4731 , piece_id=PSEQSLZZGS7HXUQNATQ4SIC2N7GVI7KNIKJLAW5NKWHKQC6GGPBQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.816449362 (548h30m4.687532397s) [noise=true , address=145.79.227.12:4767 , piece_id=YRAQOD234EKCP4NCXKHJPZJ3EGB55JZCVNOVOIRER6PPDALBCXPQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.817014498 (548h30m4.688097543s) [noise=true , address=77.90.31.12:4590 , piece_id=DW4C6EFH4VABYSDDHNWSPBDGRSGQB6NYK2N2AN4LQVOTQWO53KOA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.817897820 (548h30m4.688980855s) [noise=true , address=145.79.227.17:4972 , piece_id=EPC3AD3SOB66CKUHGVRGKQHRF7BIOJLXUHGT3E52NHDBRFZC44QA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.819095080 (548h30m4.690178115s) [noise=true , address=45.91.116.4:4937 , piece_id=NFNS4RKDARVLSPWOVNZMGSSJX6GMTXXR666SYYSWARU4J7O7TYXA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.821479180 (548h30m4.692562215s) [noise=true , address=145.79.227.52:4447 , piece_id=LJGA7XILGHRZSKYW6LNA6F2TRENZNWRGVJD2VRFWM6ZHW57INHXA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.823377093 (548h30m4.694460139s) [noise=true , address=145.79.227.45:4230 , piece_id=TYFXSONYH4NWYWGKEDDU24V3YRBJ67XRA2QZRJJ5CSEALNBPEUBQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.823712342 (548h30m4.694795387s) [noise=true , address=145.79.227.38:4977 , piece_id=RXZGLTY3C3U7YQ2M5BSMGNNQD7SNX4UPHN4BERW2CETBEAXJOVZA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.825471020 (548h30m4.696554066s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:06.536203398 (548h30m6.407286434s) [upload=480950]: Next returned nil segment Dec 28 11:59:06.536216755 (548h30m6.40729979s) [upload=480950]: Waiting for error group managing segments... Dec 28 11:59:06.817621144 (548h30m6.688704179s) [noise=true , address=145.79.227.38:4977 , piece_id=RXZGLTY3C3U7YQ2M5BSMGNNQD7SNX4UPHN4BERW2CETBEAXJOVZA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824356289 (548h30m6.695439324s) [noise=true , address=145.79.227.45:4230 , piece_id=TYFXSONYH4NWYWGKEDDU24V3YRBJ67XRA2QZRJJ5CSEALNBPEUBQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824889824 (548h30m6.695972869s) [noise=true , address=145.79.227.2:4115 , piece_id=XITRDAQHGOGTXGV577JUDTKMSGNGFRDBGT5CCNUWMOTXLYH5FEEQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824958796 (548h30m6.696041831s) [noise=true , address=145.79.227.11:4731 , piece_id=PSEQSLZZGS7HXUQNATQ4SIC2N7GVI7KNIKJLAW5NKWHKQC6GGPBQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.825405146 (548h30m6.696488181s) [noise=true , address=145.79.227.17:4972 , piece_id=EPC3AD3SOB66CKUHGVRGKQHRF7BIOJLXUHGT3E52NHDBRFZC44QA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.831799772 (548h30m6.702882808s) [noise=true , address=145.79.227.32:4836 , piece_id=3HIFE5VAZCPK7IORT6ASYBYUHUI2GSHFTVSYW7VGZZV74VNCMY4Q, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.837038968 (548h30m6.708122013s) [noise=true , address=145.79.227.52:4447 , piece_id=LJGA7XILGHRZSKYW6LNA6F2TRENZNWRGVJD2VRFWM6ZHW57INHXA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.837826047 (548h30m6.708909093s) [noise=true , address=145.79.227.16:4910 , piece_id=FXWMHX2GDSRONV35YWIB2YR3IT2ADLAW5VHSCMYIDYS4FPXS54KQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.846515072 (548h30m6.717598107s) [noise=true , address=145.79.227.12:4767 , piece_id=YRAQOD234EKCP4NCXKHJPZJ3EGB55JZCVNOVOIRER6PPDALBCXPQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.090006588 (548h30m6.961089633s) [noise=true , address=45.91.116.14:4189 , piece_id=6KJDBBVZ47CMUZ34NM5IIBFWMKQ5FZQNZZFOWJBQU2Y2DCKU4TZQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.100333844 (548h30m6.971416889s) [noise=true , address=77.90.31.12:4590 , piece_id=DW4C6EFH4VABYSDDHNWSPBDGRSGQB6NYK2N2AN4LQVOTQWO53KOA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.110463131 (548h30m6.981546166s) [noise=true , address=45.91.116.4:4937 , piece_id=NFNS4RKDARVLSPWOVNZMGSSJX6GMTXXR666SYYSWARU4J7O7TYXA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.135220584 (548h30m7.006303629s) [noise=true , address=77.90.31.11:4556 , piece_id=IONPRCI2HTCVD2UDXKWX2JFGOHYP75GR4Q6UWWQIQS7AN42OB34A, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.135575850 (548h30m7.006658895s) [seg_pos={0 0}, upload=480950]: Segment reached optimal threshold of 13 pieces. Dec 28 11:59:07.135851806 (548h30m7.006934851s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.135863117 (548h30m7.006946162s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.135890990 (548h30m7.006974025s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.135908623 (548h30m7.006991658s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.136129715 (548h30m7.00721275s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.136136227 (548h30m7.007219262s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.136155663 (548h30m7.007238699s) [seg_pos={0 0}, upload=480950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:59:07.136238061 (548h30m7.007321096s) [upload=480950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:59:07.136245014 (548h30m7.007328049s) [upload=480950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:59:07.420618777 (548h30m7.291701842s) [upload=480950]: Done uploading segments... Dec 28 11:59:07.420648463 (548h30m7.291731508s) [upload=480950]: Upload finished. err: <nil> Duration: 5.711221141s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 12.243178483s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 11:59:54 UTC) orbiter warning: context deadline exceeded Duration: 12.243205715s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX.] duration: 3.440810886s trace: https://jaeger.dev.storj.io/trace/1e623ef2e11a35bf FAIL (2025-12-28 11:59:41 UTC) orbiter warning: context deadline exceeded Duration: 3.440838419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......................XX....X.....XX...XXX.......X..XX....X.] duration: 5.712200915s trace: https://jaeger.dev.storj.io/trace/7b7bce03ce6b14d2 FAIL (2025-12-28 11:59:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:59:04.566286472 (548h30m4.437369517s) [upload=480950]: Starting upload Dec 28 11:59:04.566375662 (548h30m4.437458707s) [upload=480950]: Done starting upload Dec 28 11:59:04.566706552 (548h30m4.437789597s) [upload=480950]: Uploading segments... Dec 28 11:59:04.566777286 (548h30m4.437860331s) [upload=480950]: Got next segment. Inline: false Dec 28 11:59:04.566818064 (548h30m4.437901109s) [upload=480950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:59:04.566824617 (548h30m4.437907662s) [upload=480950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:59:04.812423396 (548h30m4.683506441s) [seg_pos={0 0}, upload=480950]: Begin upload segment... Dec 28 11:59:04.812462571 (548h30m4.683545606s) [seg_pos={0 0}, upload=480950]: Done begin upload segment. Dec 28 11:59:04.812487297 (548h30m4.683570332s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.812549786 (548h30m4.683632831s) [noise=true , address=77.90.31.11:4556 , piece_id=IONPRCI2HTCVD2UDXKWX2JFGOHYP75GR4Q6UWWQIQS7AN42OB34A, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813033688 (548h30m4.684116733s) [noise=true , address=145.79.227.2:4115 , piece_id=XITRDAQHGOGTXGV577JUDTKMSGNGFRDBGT5CCNUWMOTXLYH5FEEQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813451043 (548h30m4.684534088s) [noise=true , address=145.79.227.16:4910 , piece_id=FXWMHX2GDSRONV35YWIB2YR3IT2ADLAW5VHSCMYIDYS4FPXS54KQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813973558 (548h30m4.685056593s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.814497024 (548h30m4.685580059s) [noise=true , address=45.91.116.14:4189 , piece_id=6KJDBBVZ47CMUZ34NM5IIBFWMKQ5FZQNZZFOWJBQU2Y2DCKU4TZQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.814944406 (548h30m4.686027441s) [noise=true , address=145.79.227.32:4836 , piece_id=3HIFE5VAZCPK7IORT6ASYBYUHUI2GSHFTVSYW7VGZZV74VNCMY4Q, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.815390937 (548h30m4.686473982s) [noise=true , address=145.79.227.11:4731 , piece_id=PSEQSLZZGS7HXUQNATQ4SIC2N7GVI7KNIKJLAW5NKWHKQC6GGPBQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.816449362 (548h30m4.687532397s) [noise=true , address=145.79.227.12:4767 , piece_id=YRAQOD234EKCP4NCXKHJPZJ3EGB55JZCVNOVOIRER6PPDALBCXPQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.817014498 (548h30m4.688097543s) [noise=true , address=77.90.31.12:4590 , piece_id=DW4C6EFH4VABYSDDHNWSPBDGRSGQB6NYK2N2AN4LQVOTQWO53KOA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.817897820 (548h30m4.688980855s) [noise=true , address=145.79.227.17:4972 , piece_id=EPC3AD3SOB66CKUHGVRGKQHRF7BIOJLXUHGT3E52NHDBRFZC44QA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.819095080 (548h30m4.690178115s) [noise=true , address=45.91.116.4:4937 , piece_id=NFNS4RKDARVLSPWOVNZMGSSJX6GMTXXR666SYYSWARU4J7O7TYXA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.821479180 (548h30m4.692562215s) [noise=true , address=145.79.227.52:4447 , piece_id=LJGA7XILGHRZSKYW6LNA6F2TRENZNWRGVJD2VRFWM6ZHW57INHXA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.823377093 (548h30m4.694460139s) [noise=true , address=145.79.227.45:4230 , piece_id=TYFXSONYH4NWYWGKEDDU24V3YRBJ67XRA2QZRJJ5CSEALNBPEUBQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.823712342 (548h30m4.694795387s) [noise=true , address=145.79.227.38:4977 , piece_id=RXZGLTY3C3U7YQ2M5BSMGNNQD7SNX4UPHN4BERW2CETBEAXJOVZA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.825471020 (548h30m4.696554066s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:06.536203398 (548h30m6.407286434s) [upload=480950]: Next returned nil segment Dec 28 11:59:06.536216755 (548h30m6.40729979s) [upload=480950]: Waiting for error group managing segments... Dec 28 11:59:06.817621144 (548h30m6.688704179s) [noise=true , address=145.79.227.38:4977 , piece_id=RXZGLTY3C3U7YQ2M5BSMGNNQD7SNX4UPHN4BERW2CETBEAXJOVZA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824356289 (548h30m6.695439324s) [noise=true , address=145.79.227.45:4230 , piece_id=TYFXSONYH4NWYWGKEDDU24V3YRBJ67XRA2QZRJJ5CSEALNBPEUBQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824889824 (548h30m6.695972869s) [noise=true , address=145.79.227.2:4115 , piece_id=XITRDAQHGOGTXGV577JUDTKMSGNGFRDBGT5CCNUWMOTXLYH5FEEQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824958796 (548h30m6.696041831s) [noise=true , address=145.79.227.11:4731 , piece_id=PSEQSLZZGS7HXUQNATQ4SIC2N7GVI7KNIKJLAW5NKWHKQC6GGPBQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.825405146 (548h30m6.696488181s) [noise=true , address=145.79.227.17:4972 , piece_id=EPC3AD3SOB66CKUHGVRGKQHRF7BIOJLXUHGT3E52NHDBRFZC44QA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.831799772 (548h30m6.702882808s) [noise=true , address=145.79.227.32:4836 , piece_id=3HIFE5VAZCPK7IORT6ASYBYUHUI2GSHFTVSYW7VGZZV74VNCMY4Q, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.837038968 (548h30m6.708122013s) [noise=true , address=145.79.227.52:4447 , piece_id=LJGA7XILGHRZSKYW6LNA6F2TRENZNWRGVJD2VRFWM6ZHW57INHXA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.837826047 (548h30m6.708909093s) [noise=true , address=145.79.227.16:4910 , piece_id=FXWMHX2GDSRONV35YWIB2YR3IT2ADLAW5VHSCMYIDYS4FPXS54KQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.846515072 (548h30m6.717598107s) [noise=true , address=145.79.227.12:4767 , piece_id=YRAQOD234EKCP4NCXKHJPZJ3EGB55JZCVNOVOIRER6PPDALBCXPQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.090006588 (548h30m6.961089633s) [noise=true , address=45.91.116.14:4189 , piece_id=6KJDBBVZ47CMUZ34NM5IIBFWMKQ5FZQNZZFOWJBQU2Y2DCKU4TZQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.100333844 (548h30m6.971416889s) [noise=true , address=77.90.31.12:4590 , piece_id=DW4C6EFH4VABYSDDHNWSPBDGRSGQB6NYK2N2AN4LQVOTQWO53KOA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.110463131 (548h30m6.981546166s) [noise=true , address=45.91.116.4:4937 , piece_id=NFNS4RKDARVLSPWOVNZMGSSJX6GMTXXR666SYYSWARU4J7O7TYXA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.135220584 (548h30m7.006303629s) [noise=true , address=77.90.31.11:4556 , piece_id=IONPRCI2HTCVD2UDXKWX2JFGOHYP75GR4Q6UWWQIQS7AN42OB34A, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.135575850 (548h30m7.006658895s) [seg_pos={0 0}, upload=480950]: Segment reached optimal threshold of 13 pieces. Dec 28 11:59:07.135851806 (548h30m7.006934851s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.135863117 (548h30m7.006946162s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.135890990 (548h30m7.006974025s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.135908623 (548h30m7.006991658s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.136129715 (548h30m7.00721275s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.136136227 (548h30m7.007219262s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.136155663 (548h30m7.007238699s) [seg_pos={0 0}, upload=480950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:59:07.136238061 (548h30m7.007321096s) [upload=480950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:59:07.136245014 (548h30m7.007328049s) [upload=480950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:59:07.420618777 (548h30m7.291701842s) [upload=480950]: Done uploading segments... Dec 28 11:59:07.420648463 (548h30m7.291731508s) [upload=480950]: Upload finished. err: <nil> Duration: 5.712217746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......................XX....X.....XX...XXX.......X..XX....X] duration: 30.003972372s trace: https://jaeger.dev.storj.io/trace/7b7bce03ce6b14d2 FAIL (2025-12-28 11:59:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:59:04.566286472 (548h30m4.437369517s) [upload=480950]: Starting upload Dec 28 11:59:04.566375662 (548h30m4.437458707s) [upload=480950]: Done starting upload Dec 28 11:59:04.566706552 (548h30m4.437789597s) [upload=480950]: Uploading segments... Dec 28 11:59:04.566777286 (548h30m4.437860331s) [upload=480950]: Got next segment. Inline: false Dec 28 11:59:04.566818064 (548h30m4.437901109s) [upload=480950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:59:04.566824617 (548h30m4.437907662s) [upload=480950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:59:04.812423396 (548h30m4.683506441s) [seg_pos={0 0}, upload=480950]: Begin upload segment... Dec 28 11:59:04.812462571 (548h30m4.683545606s) [seg_pos={0 0}, upload=480950]: Done begin upload segment. Dec 28 11:59:04.812487297 (548h30m4.683570332s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.812549786 (548h30m4.683632831s) [noise=true , address=77.90.31.11:4556 , piece_id=IONPRCI2HTCVD2UDXKWX2JFGOHYP75GR4Q6UWWQIQS7AN42OB34A, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813033688 (548h30m4.684116733s) [noise=true , address=145.79.227.2:4115 , piece_id=XITRDAQHGOGTXGV577JUDTKMSGNGFRDBGT5CCNUWMOTXLYH5FEEQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813451043 (548h30m4.684534088s) [noise=true , address=145.79.227.16:4910 , piece_id=FXWMHX2GDSRONV35YWIB2YR3IT2ADLAW5VHSCMYIDYS4FPXS54KQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.813973558 (548h30m4.685056593s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.814497024 (548h30m4.685580059s) [noise=true , address=45.91.116.14:4189 , piece_id=6KJDBBVZ47CMUZ34NM5IIBFWMKQ5FZQNZZFOWJBQU2Y2DCKU4TZQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.814944406 (548h30m4.686027441s) [noise=true , address=145.79.227.32:4836 , piece_id=3HIFE5VAZCPK7IORT6ASYBYUHUI2GSHFTVSYW7VGZZV74VNCMY4Q, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.815390937 (548h30m4.686473982s) [noise=true , address=145.79.227.11:4731 , piece_id=PSEQSLZZGS7HXUQNATQ4SIC2N7GVI7KNIKJLAW5NKWHKQC6GGPBQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.816449362 (548h30m4.687532397s) [noise=true , address=145.79.227.12:4767 , piece_id=YRAQOD234EKCP4NCXKHJPZJ3EGB55JZCVNOVOIRER6PPDALBCXPQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.817014498 (548h30m4.688097543s) [noise=true , address=77.90.31.12:4590 , piece_id=DW4C6EFH4VABYSDDHNWSPBDGRSGQB6NYK2N2AN4LQVOTQWO53KOA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.817897820 (548h30m4.688980855s) [noise=true , address=145.79.227.17:4972 , piece_id=EPC3AD3SOB66CKUHGVRGKQHRF7BIOJLXUHGT3E52NHDBRFZC44QA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.819095080 (548h30m4.690178115s) [noise=true , address=45.91.116.4:4937 , piece_id=NFNS4RKDARVLSPWOVNZMGSSJX6GMTXXR666SYYSWARU4J7O7TYXA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.821479180 (548h30m4.692562215s) [noise=true , address=145.79.227.52:4447 , piece_id=LJGA7XILGHRZSKYW6LNA6F2TRENZNWRGVJD2VRFWM6ZHW57INHXA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.823377093 (548h30m4.694460139s) [noise=true , address=145.79.227.45:4230 , piece_id=TYFXSONYH4NWYWGKEDDU24V3YRBJ67XRA2QZRJJ5CSEALNBPEUBQ, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.823712342 (548h30m4.694795387s) [noise=true , address=145.79.227.38:4977 , piece_id=RXZGLTY3C3U7YQ2M5BSMGNNQD7SNX4UPHN4BERW2CETBEAXJOVZA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:04.825471020 (548h30m4.696554066s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Uploading piece... Dec 28 11:59:06.536203398 (548h30m6.407286434s) [upload=480950]: Next returned nil segment Dec 28 11:59:06.536216755 (548h30m6.40729979s) [upload=480950]: Waiting for error group managing segments... Dec 28 11:59:06.817621144 (548h30m6.688704179s) [noise=true , address=145.79.227.38:4977 , piece_id=RXZGLTY3C3U7YQ2M5BSMGNNQD7SNX4UPHN4BERW2CETBEAXJOVZA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824356289 (548h30m6.695439324s) [noise=true , address=145.79.227.45:4230 , piece_id=TYFXSONYH4NWYWGKEDDU24V3YRBJ67XRA2QZRJJ5CSEALNBPEUBQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824889824 (548h30m6.695972869s) [noise=true , address=145.79.227.2:4115 , piece_id=XITRDAQHGOGTXGV577JUDTKMSGNGFRDBGT5CCNUWMOTXLYH5FEEQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.824958796 (548h30m6.696041831s) [noise=true , address=145.79.227.11:4731 , piece_id=PSEQSLZZGS7HXUQNATQ4SIC2N7GVI7KNIKJLAW5NKWHKQC6GGPBQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.825405146 (548h30m6.696488181s) [noise=true , address=145.79.227.17:4972 , piece_id=EPC3AD3SOB66CKUHGVRGKQHRF7BIOJLXUHGT3E52NHDBRFZC44QA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.831799772 (548h30m6.702882808s) [noise=true , address=145.79.227.32:4836 , piece_id=3HIFE5VAZCPK7IORT6ASYBYUHUI2GSHFTVSYW7VGZZV74VNCMY4Q, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.837038968 (548h30m6.708122013s) [noise=true , address=145.79.227.52:4447 , piece_id=LJGA7XILGHRZSKYW6LNA6F2TRENZNWRGVJD2VRFWM6ZHW57INHXA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.837826047 (548h30m6.708909093s) [noise=true , address=145.79.227.16:4910 , piece_id=FXWMHX2GDSRONV35YWIB2YR3IT2ADLAW5VHSCMYIDYS4FPXS54KQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:06.846515072 (548h30m6.717598107s) [noise=true , address=145.79.227.12:4767 , piece_id=YRAQOD234EKCP4NCXKHJPZJ3EGB55JZCVNOVOIRER6PPDALBCXPQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.090006588 (548h30m6.961089633s) [noise=true , address=45.91.116.14:4189 , piece_id=6KJDBBVZ47CMUZ34NM5IIBFWMKQ5FZQNZZFOWJBQU2Y2DCKU4TZQ, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.100333844 (548h30m6.971416889s) [noise=true , address=77.90.31.12:4590 , piece_id=DW4C6EFH4VABYSDDHNWSPBDGRSGQB6NYK2N2AN4LQVOTQWO53KOA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.110463131 (548h30m6.981546166s) [noise=true , address=45.91.116.4:4937 , piece_id=NFNS4RKDARVLSPWOVNZMGSSJX6GMTXXR666SYYSWARU4J7O7TYXA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.135220584 (548h30m7.006303629s) [noise=true , address=77.90.31.11:4556 , piece_id=IONPRCI2HTCVD2UDXKWX2JFGOHYP75GR4Q6UWWQIQS7AN42OB34A, seg_pos={0 0}, upload=480950]: Done uploading piece. err: <nil> Dec 28 11:59:07.135575850 (548h30m7.006658895s) [seg_pos={0 0}, upload=480950]: Segment reached optimal threshold of 13 pieces. Dec 28 11:59:07.135851806 (548h30m7.006934851s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.135863117 (548h30m7.006946162s) [noise=true , address=77.90.31.17:4770 , piece_id=N4SLDGTFZYVVEWNKAPYRX4ICO4CAVPDSOYSSZYANM3LQ5PEVPDPA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.135890990 (548h30m7.006974025s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.135908623 (548h30m7.006991658s) [noise=true , address=45.91.116.10:4082 , piece_id=UX6E7XMBO6HUWTR2RH7UW5IDS3SUGRG4GMAJQ45HJVKAHQRDOFIA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.136129715 (548h30m7.00721275s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 11:59:07.136136227 (548h30m7.007219262s) [noise=true , address=45.91.116.8:4048 , piece_id=PXKWNMT3JANPU7FNOSAXVXIUTVKQ3DQLZLPA7OAOXJFO2V3Z2URA, seg_pos={0 0}, upload=480950]: Long tail cancellation due to optimal threshold Dec 28 11:59:07.136155663 (548h30m7.007238699s) [seg_pos={0 0}, upload=480950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:59:07.136238061 (548h30m7.007321096s) [upload=480950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:59:07.136245014 (548h30m7.007328049s) [upload=480950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:59:07.420618777 (548h30m7.291701842s) [upload=480950]: Done uploading segments... Dec 28 11:59:07.420648463 (548h30m7.291731508s) [upload=480950]: Upload finished. err: <nil> Duration: 30.003998091s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.001674578s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 11:59:26 UTC) orbiter warning: context deadline exceeded Duration: 30.001697461s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...........X..X....................................] duration: 35.207066107s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:59:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 35.207082898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X........X...........X..X...................................] duration: 38.700027897s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:58:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 38.700060097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX] duration: 5.001224505s trace: https://jaeger.dev.storj.io/trace/1e623ef2e11a35bf FAIL (2025-12-28 11:58:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001296732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.769707223s trace: https://jaeger.dev.storj.io/trace/5b27cb814538279c FAIL (2025-12-28 11:58:26 UTC) orbiter warning: context deadline exceeded Duration: 14.769730185s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.......................XX....X.....XX...XXX.......X..XX....] duration: 5.812960019s trace: https://jaeger.dev.storj.io/trace/10485f0d6e7d8b7b FAIL (2025-12-28 11:58:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA) context deadline exceeded ======LOGS====== 11:55:55.221684426 (548h26m55.092767461s) [upload=480904]: Starting upload Dec 28 11:55:55.221742307 (548h26m55.092825342s) [upload=480904]: Done starting upload Dec 28 11:55:55.221971212 (548h26m55.093054257s) [upload=480904]: Uploading segments... Dec 28 11:55:55.222010907 (548h26m55.093093942s) [upload=480904]: Got next segment. Inline: false Dec 28 11:55:55.222034953 (548h26m55.093117988s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:55:55.222038911 (548h26m55.093121956s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:55:55.468718427 (548h26m55.339801472s) [seg_pos={0 0}, upload=480904]: Begin upload segment... Dec 28 11:55:55.468772049 (548h26m55.339855085s) [seg_pos={0 0}, upload=480904]: Done begin upload segment. Dec 28 11:55:55.468840830 (548h26m55.339923865s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.468847513 (548h26m55.339930548s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469468265 (548h26m55.34055131s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469544841 (548h26m55.340627886s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469878126 (548h26m55.340961161s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470146416 (548h26m55.341229451s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470521791 (548h26m55.341604836s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470814128 (548h26m55.341897163s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471047823 (548h26m55.342130858s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471313409 (548h26m55.342396454s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471614262 (548h26m55.342697307s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471757645 (548h26m55.34284071s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471887782 (548h26m55.342970817s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472509566 (548h26m55.343592601s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472984441 (548h26m55.344067486s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.473535911 (548h26m55.344618956s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:57.087410286 (548h26m56.958493321s) [upload=480904]: Next returned nil segment Dec 28 11:55:57.087424163 (548h26m56.958507198s) [upload=480904]: Waiting for error group managing segments... Dec 28 11:55:57.472670363 (548h26m57.343753398s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.478174193 (548h26m57.349257239s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.479751196 (548h26m57.350834241s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.482687277 (548h26m57.353770323s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.484092894 (548h26m57.355175929s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494765497 (548h26m57.365848542s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494975587 (548h26m57.366058622s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.498947889 (548h26m57.370030924s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.499149724 (548h26m57.370232769s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507234589 (548h26m57.378317624s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507952787 (548h26m57.379035822s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.668360088 (548h26m57.539443123s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691349503 (548h26m57.562432538s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691370223 (548h26m57.562453258s) [seg_pos={0 0}, upload=480904]: Segment reached optimal threshold of 13 pieces. Dec 28 11:55:57.691526902 (548h26m57.562609947s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection Dec 28 11:55:57.691534516 (548h26m57.562617551s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691673180 (548h26m57.562756215s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691682979 (548h26m57.562766024s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691694801 (548h26m57.562777856s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691702386 (548h26m57.562785421s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691803027 (548h26m57.562886062s) [seg_pos={0 0}, upload=480904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:55:57.691914931 (548h26m57.562997966s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:55:57.691923747 (548h26m57.563006782s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:55:57.979417271 (548h26m57.850500316s) [upload=480904]: Done uploading segments... Dec 28 11:55:57.979449572 (548h26m57.850532607s) [upload=480904]: Upload finished. err: <nil> Duration: 5.81298713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.......................XX....X.....XX...XXX.......X..XX...] duration: 5.719559648s trace: https://jaeger.dev.storj.io/trace/10485f0d6e7d8b7b FAIL (2025-12-28 11:57:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA) context deadline exceeded ======LOGS====== 11:55:55.221684426 (548h26m55.092767461s) [upload=480904]: Starting upload Dec 28 11:55:55.221742307 (548h26m55.092825342s) [upload=480904]: Done starting upload Dec 28 11:55:55.221971212 (548h26m55.093054257s) [upload=480904]: Uploading segments... Dec 28 11:55:55.222010907 (548h26m55.093093942s) [upload=480904]: Got next segment. Inline: false Dec 28 11:55:55.222034953 (548h26m55.093117988s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:55:55.222038911 (548h26m55.093121956s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:55:55.468718427 (548h26m55.339801472s) [seg_pos={0 0}, upload=480904]: Begin upload segment... Dec 28 11:55:55.468772049 (548h26m55.339855085s) [seg_pos={0 0}, upload=480904]: Done begin upload segment. Dec 28 11:55:55.468840830 (548h26m55.339923865s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.468847513 (548h26m55.339930548s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469468265 (548h26m55.34055131s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469544841 (548h26m55.340627886s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469878126 (548h26m55.340961161s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470146416 (548h26m55.341229451s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470521791 (548h26m55.341604836s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470814128 (548h26m55.341897163s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471047823 (548h26m55.342130858s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471313409 (548h26m55.342396454s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471614262 (548h26m55.342697307s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471757645 (548h26m55.34284071s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471887782 (548h26m55.342970817s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472509566 (548h26m55.343592601s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472984441 (548h26m55.344067486s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.473535911 (548h26m55.344618956s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:57.087410286 (548h26m56.958493321s) [upload=480904]: Next returned nil segment Dec 28 11:55:57.087424163 (548h26m56.958507198s) [upload=480904]: Waiting for error group managing segments... Dec 28 11:55:57.472670363 (548h26m57.343753398s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.478174193 (548h26m57.349257239s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.479751196 (548h26m57.350834241s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.482687277 (548h26m57.353770323s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.484092894 (548h26m57.355175929s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494765497 (548h26m57.365848542s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494975587 (548h26m57.366058622s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.498947889 (548h26m57.370030924s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.499149724 (548h26m57.370232769s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507234589 (548h26m57.378317624s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507952787 (548h26m57.379035822s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.668360088 (548h26m57.539443123s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691349503 (548h26m57.562432538s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691370223 (548h26m57.562453258s) [seg_pos={0 0}, upload=480904]: Segment reached optimal threshold of 13 pieces. Dec 28 11:55:57.691526902 (548h26m57.562609947s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection Dec 28 11:55:57.691534516 (548h26m57.562617551s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691673180 (548h26m57.562756215s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691682979 (548h26m57.562766024s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691694801 (548h26m57.562777856s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691702386 (548h26m57.562785421s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691803027 (548h26m57.562886062s) [seg_pos={0 0}, upload=480904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:55:57.691914931 (548h26m57.562997966s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:55:57.691923747 (548h26m57.563006782s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:55:57.979417271 (548h26m57.850500316s) [upload=480904]: Done uploading segments... Dec 28 11:55:57.979449572 (548h26m57.850532607s) [upload=480904]: Upload finished. err: <nil> Duration: 5.719587702s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXX] duration: 5.00230447s trace: https://jaeger.dev.storj.io/trace/c9c744c95437ae9 FAIL (2025-12-28 11:57:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002329226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X........X...........X..X..................................] duration: 26.439132492s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:57:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 26.439151558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000883506s trace: https://jaeger.dev.storj.io/trace/5b27cb814538279c FAIL (2025-12-28 11:57:33 UTC) orbiter warning: context deadline exceeded Duration: 30.000907192s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.......................XX....X.....XX...XXX.......X..XX..] duration: 5.702659037s trace: https://jaeger.dev.storj.io/trace/10485f0d6e7d8b7b FAIL (2025-12-28 11:57:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA) context deadline exceeded ======LOGS====== 11:55:55.221684426 (548h26m55.092767461s) [upload=480904]: Starting upload Dec 28 11:55:55.221742307 (548h26m55.092825342s) [upload=480904]: Done starting upload Dec 28 11:55:55.221971212 (548h26m55.093054257s) [upload=480904]: Uploading segments... Dec 28 11:55:55.222010907 (548h26m55.093093942s) [upload=480904]: Got next segment. Inline: false Dec 28 11:55:55.222034953 (548h26m55.093117988s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:55:55.222038911 (548h26m55.093121956s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:55:55.468718427 (548h26m55.339801472s) [seg_pos={0 0}, upload=480904]: Begin upload segment... Dec 28 11:55:55.468772049 (548h26m55.339855085s) [seg_pos={0 0}, upload=480904]: Done begin upload segment. Dec 28 11:55:55.468840830 (548h26m55.339923865s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.468847513 (548h26m55.339930548s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469468265 (548h26m55.34055131s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469544841 (548h26m55.340627886s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469878126 (548h26m55.340961161s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470146416 (548h26m55.341229451s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470521791 (548h26m55.341604836s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470814128 (548h26m55.341897163s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471047823 (548h26m55.342130858s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471313409 (548h26m55.342396454s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471614262 (548h26m55.342697307s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471757645 (548h26m55.34284071s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471887782 (548h26m55.342970817s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472509566 (548h26m55.343592601s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472984441 (548h26m55.344067486s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.473535911 (548h26m55.344618956s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:57.087410286 (548h26m56.958493321s) [upload=480904]: Next returned nil segment Dec 28 11:55:57.087424163 (548h26m56.958507198s) [upload=480904]: Waiting for error group managing segments... Dec 28 11:55:57.472670363 (548h26m57.343753398s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.478174193 (548h26m57.349257239s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.479751196 (548h26m57.350834241s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.482687277 (548h26m57.353770323s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.484092894 (548h26m57.355175929s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494765497 (548h26m57.365848542s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494975587 (548h26m57.366058622s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.498947889 (548h26m57.370030924s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.499149724 (548h26m57.370232769s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507234589 (548h26m57.378317624s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507952787 (548h26m57.379035822s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.668360088 (548h26m57.539443123s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691349503 (548h26m57.562432538s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691370223 (548h26m57.562453258s) [seg_pos={0 0}, upload=480904]: Segment reached optimal threshold of 13 pieces. Dec 28 11:55:57.691526902 (548h26m57.562609947s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection Dec 28 11:55:57.691534516 (548h26m57.562617551s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691673180 (548h26m57.562756215s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691682979 (548h26m57.562766024s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691694801 (548h26m57.562777856s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691702386 (548h26m57.562785421s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691803027 (548h26m57.562886062s) [seg_pos={0 0}, upload=480904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:55:57.691914931 (548h26m57.562997966s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:55:57.691923747 (548h26m57.563006782s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:55:57.979417271 (548h26m57.850500316s) [upload=480904]: Done uploading segments... Dec 28 11:55:57.979449572 (548h26m57.850532607s) [upload=480904]: Upload finished. err: <nil> Duration: 5.702685779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X........X...........X..X.................................] duration: 42.114492513s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:57:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 42.114508954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXX] duration: 5.001233851s trace: https://jaeger.dev.storj.io/trace/6e775e0144f49e3d FAIL (2025-12-28 11:57:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001262326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.......................XX....X.....XX...XXX.......X..XX.] duration: 5.674044702s trace: https://jaeger.dev.storj.io/trace/10485f0d6e7d8b7b FAIL (2025-12-28 11:56:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA) context deadline exceeded ======LOGS====== 11:55:55.221684426 (548h26m55.092767461s) [upload=480904]: Starting upload Dec 28 11:55:55.221742307 (548h26m55.092825342s) [upload=480904]: Done starting upload Dec 28 11:55:55.221971212 (548h26m55.093054257s) [upload=480904]: Uploading segments... Dec 28 11:55:55.222010907 (548h26m55.093093942s) [upload=480904]: Got next segment. Inline: false Dec 28 11:55:55.222034953 (548h26m55.093117988s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:55:55.222038911 (548h26m55.093121956s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:55:55.468718427 (548h26m55.339801472s) [seg_pos={0 0}, upload=480904]: Begin upload segment... Dec 28 11:55:55.468772049 (548h26m55.339855085s) [seg_pos={0 0}, upload=480904]: Done begin upload segment. Dec 28 11:55:55.468840830 (548h26m55.339923865s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.468847513 (548h26m55.339930548s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469468265 (548h26m55.34055131s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469544841 (548h26m55.340627886s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469878126 (548h26m55.340961161s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470146416 (548h26m55.341229451s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470521791 (548h26m55.341604836s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470814128 (548h26m55.341897163s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471047823 (548h26m55.342130858s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471313409 (548h26m55.342396454s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471614262 (548h26m55.342697307s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471757645 (548h26m55.34284071s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471887782 (548h26m55.342970817s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472509566 (548h26m55.343592601s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472984441 (548h26m55.344067486s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.473535911 (548h26m55.344618956s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:57.087410286 (548h26m56.958493321s) [upload=480904]: Next returned nil segment Dec 28 11:55:57.087424163 (548h26m56.958507198s) [upload=480904]: Waiting for error group managing segments... Dec 28 11:55:57.472670363 (548h26m57.343753398s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.478174193 (548h26m57.349257239s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.479751196 (548h26m57.350834241s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.482687277 (548h26m57.353770323s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.484092894 (548h26m57.355175929s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494765497 (548h26m57.365848542s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494975587 (548h26m57.366058622s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.498947889 (548h26m57.370030924s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.499149724 (548h26m57.370232769s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507234589 (548h26m57.378317624s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507952787 (548h26m57.379035822s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.668360088 (548h26m57.539443123s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691349503 (548h26m57.562432538s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691370223 (548h26m57.562453258s) [seg_pos={0 0}, upload=480904]: Segment reached optimal threshold of 13 pieces. Dec 28 11:55:57.691526902 (548h26m57.562609947s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection Dec 28 11:55:57.691534516 (548h26m57.562617551s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691673180 (548h26m57.562756215s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691682979 (548h26m57.562766024s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691694801 (548h26m57.562777856s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691702386 (548h26m57.562785421s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691803027 (548h26m57.562886062s) [seg_pos={0 0}, upload=480904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:55:57.691914931 (548h26m57.562997966s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:55:57.691923747 (548h26m57.563006782s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:55:57.979417271 (548h26m57.850500316s) [upload=480904]: Done uploading segments... Dec 28 11:55:57.979449572 (548h26m57.850532607s) [upload=480904]: Upload finished. err: <nil> Duration: 5.674062347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.......................XX....X.....XX...XXX.......X..XX] duration: 30.002594964s trace: https://jaeger.dev.storj.io/trace/10485f0d6e7d8b7b FAIL (2025-12-28 11:56:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA) context deadline exceeded ======LOGS====== 11:55:55.221684426 (548h26m55.092767461s) [upload=480904]: Starting upload Dec 28 11:55:55.221742307 (548h26m55.092825342s) [upload=480904]: Done starting upload Dec 28 11:55:55.221971212 (548h26m55.093054257s) [upload=480904]: Uploading segments... Dec 28 11:55:55.222010907 (548h26m55.093093942s) [upload=480904]: Got next segment. Inline: false Dec 28 11:55:55.222034953 (548h26m55.093117988s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:55:55.222038911 (548h26m55.093121956s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:55:55.468718427 (548h26m55.339801472s) [seg_pos={0 0}, upload=480904]: Begin upload segment... Dec 28 11:55:55.468772049 (548h26m55.339855085s) [seg_pos={0 0}, upload=480904]: Done begin upload segment. Dec 28 11:55:55.468840830 (548h26m55.339923865s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.468847513 (548h26m55.339930548s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469468265 (548h26m55.34055131s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469544841 (548h26m55.340627886s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.469878126 (548h26m55.340961161s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470146416 (548h26m55.341229451s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470521791 (548h26m55.341604836s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.470814128 (548h26m55.341897163s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471047823 (548h26m55.342130858s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471313409 (548h26m55.342396454s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471614262 (548h26m55.342697307s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471757645 (548h26m55.34284071s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.471887782 (548h26m55.342970817s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472509566 (548h26m55.343592601s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.472984441 (548h26m55.344067486s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:55.473535911 (548h26m55.344618956s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Uploading piece... Dec 28 11:55:57.087410286 (548h26m56.958493321s) [upload=480904]: Next returned nil segment Dec 28 11:55:57.087424163 (548h26m56.958507198s) [upload=480904]: Waiting for error group managing segments... Dec 28 11:55:57.472670363 (548h26m57.343753398s) [noise=true , address=145.79.227.3:4151 , piece_id=35X7SIL7MYSHXOKZQZKDSLKYU7ZIFXLJZWEWM5LNZMNTYBBOAKFQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.478174193 (548h26m57.349257239s) [noise=true , address=145.79.227.16:4911 , piece_id=HNG53E5ELSIIQT2LZNIU27KNOAZPLPFY7AUNCFVQF7ZLHRLTD7QQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.479751196 (548h26m57.350834241s) [noise=true , address=145.79.227.42:4123 , piece_id=76LNV35GSHKBG2NNPIKDH5HJQENOSIUKDA2HTVXEAEXPQYLJWKIA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.482687277 (548h26m57.353770323s) [noise=true , address=145.79.227.41:4086 , piece_id=ZZLIPHEIETZAHWST22WPEKKTFRCHSURHMVEVH5NAZSWODUT3TB5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.484092894 (548h26m57.355175929s) [noise=true , address=145.79.227.4:4185 , piece_id=MTY22TTL63FA7SU6IYQ6RHE2HH4JRGUM4L25WMFJXK2NUIQE5YBA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494765497 (548h26m57.365848542s) [noise=true , address=145.79.227.32:4836 , piece_id=SRG4RDVZXFDMROPZYIMNMMGF3DG2SDMFF7RO3YJERAPRLKBP3ATQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.494975587 (548h26m57.366058622s) [noise=true , address=145.79.227.2:4114 , piece_id=LB6E3FR227IIRKKGYQJVSESR7GGVJG4SIMW5BDQ6HXJDB6JJUHLA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.498947889 (548h26m57.370030924s) [noise=true , address=145.79.227.15:4873 , piece_id=I4RWFY3KGLBALBFOO7MRXJDKGD5BGACM4LPO3KGIEXTCPC7JPWGA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.499149724 (548h26m57.370232769s) [noise=true , address=145.79.227.17:4971 , piece_id=37NWMKDPRJAV4DVK7VK3KUG5FMHXZR6INKJBEYYUCHPFELITES4A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507234589 (548h26m57.378317624s) [noise=true , address=145.79.227.37:4567 , piece_id=QDMDA77MD253KLLUHNUKA7SAW3XFPFMAOFJD6LSDGNJ56GYUCZDQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.507952787 (548h26m57.379035822s) [noise=true , address=145.79.227.36:4518 , piece_id=COVTV6IOISO6PXCVGCG4FUQZ4JM5EGHBVQPXXNRUPJ6OK4J45XPA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.668360088 (548h26m57.539443123s) [noise=true , address=77.90.31.21:4877 , piece_id=YH7H4U4SNOMDBJXR4FLILMPNRXINCEH26XEM5S2WVYKC4GFHUPRQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691349503 (548h26m57.562432538s) [noise=true , address=45.91.116.8:4047 , piece_id=TFZAWZ6EHW462RSQU5JVU2XE7ABIHZDUDDVM4UMBZJ6XEVML3CZA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: <nil> Dec 28 11:55:57.691370223 (548h26m57.562453258s) [seg_pos={0 0}, upload=480904]: Segment reached optimal threshold of 13 pieces. Dec 28 11:55:57.691526902 (548h26m57.562609947s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45412->45.91.116.9:4261: use of closed network connection Dec 28 11:55:57.691534516 (548h26m57.562617551s) [noise=true , address=45.91.116.9:4261 , piece_id=YOL5YJWGGS5EWPSJD3FT45ZOHXFJ42ORLBIEBJLSBTA3SKGOYUAQ, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691673180 (548h26m57.562756215s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691682979 (548h26m57.562766024s) [noise=true , address=45.91.116.4:4939 , piece_id=MOL2JHARDTBYAXGEC3M72ZK2AVOIGKUFDJCX6XRPVJ6T245H2L5A, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691694801 (548h26m57.562777856s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 11:55:57.691702386 (548h26m57.562785421s) [noise=true , address=145.79.227.40:4050 , piece_id=JW6IR7PM4FHTFK6UFNEMBJT53NF6JZBJH4GMQJ5FPNDXFTCYZKSA, seg_pos={0 0}, upload=480904]: Long tail cancellation due to optimal threshold Dec 28 11:55:57.691803027 (548h26m57.562886062s) [seg_pos={0 0}, upload=480904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:55:57.691914931 (548h26m57.562997966s) [upload=480904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:55:57.691923747 (548h26m57.563006782s) [upload=480904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:55:57.979417271 (548h26m57.850500316s) [upload=480904]: Done uploading segments... Dec 28 11:55:57.979449572 (548h26m57.850532607s) [upload=480904]: Upload finished. err: <nil> Duration: 30.002785958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X........X...........X..X................................] duration: 47.214771114s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:56:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 47.214792998s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXX] duration: 5.001441888s trace: https://jaeger.dev.storj.io/trace/6f3dbfb5e7fbffc2 FAIL (2025-12-28 11:56:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001458027s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXX] duration: 5.001571304s trace: https://jaeger.dev.storj.io/trace/5009d4a2b2a1be21 FAIL (2025-12-28 11:55:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00159632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X........X...........X..X...............................] duration: 33.639243492s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:55:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 33.639277788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.......................XX....X.....XX...XXX.......X..X] duration: 30.003775325s trace: https://jaeger.dev.storj.io/trace/43ad848d4052a38a FAIL (2025-12-28 11:55:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: YMROM3S4LAMKIFHXUHCVILQJVCCOTIVIKPCF5A6BZ4XQOWDM2X2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: APT5L7ZGKZK6EN3KQYTDMFJJWJPZOIEW6UHF3VL6K43VTBIEEXVQ) context deadline exceeded ======LOGS====== Dec 28 11:54:45.989367431 (548h25m45.860450476s) [upload=480885]: Starting upload Dec 28 11:54:45.989445410 (548h25m45.860528485s) [upload=480885]: Done starting upload Dec 28 11:54:45.989679565 (548h25m45.8607626s) [upload=480885]: Uploading segments... Dec 28 11:54:45.989761441 (548h25m45.860844487s) [upload=480885]: Got next segment. Inline: false Dec 28 11:54:45.989844179 (548h25m45.860927224s) [upload=480885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:54:45.989850901 (548h25m45.860933956s) [upload=480885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:54:46.230141784 (548h25m46.101224819s) [seg_pos={0 0}, upload=480885]: Begin upload segment... Dec 28 11:54:46.230194134 (548h25m46.101277169s) [seg_pos={0 0}, upload=480885]: Done begin upload segment. Dec 28 11:54:46.230411748 (548h25m46.101494783s) [noise=true , address=45.91.116.7:4012 , piece_id=4SQMNLDA6Y775WYSFWSPHQB6TOZANLUSK4M4OWPDJ3YQM277ORJA, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.230456212 (548h25m46.101539248s) [noise=true , address=45.91.116.4:4939 , piece_id=2LFWZM3YTSMC4EVQF54TFONZCOLGPI5XP6EYKB4IGFRLJN2NM2QA, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.231045765 (548h25m46.1021288s) [noise=true , address=45.91.116.11:4120 , piece_id=YMROM3S4LAMKIFHXUHCVILQJVCCOTIVIKPCF5A6BZ4XQOWDM2X2A, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.231403877 (548h25m46.102486912s) [noise=true , address=145.79.227.48:4302 , piece_id=APT5L7ZGKZK6EN3KQYTDMFJJWJPZOIEW6UHF3VL6K43VTBIEEXVQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.231672469 (548h25m46.102755504s) [noise=true , address=145.79.227.36:4517 , piece_id=N4XA2FIFHVSORQZWLTEI3O6RBD2JGG7WOMQWP7N6YIXKJJBYTCVA, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.234046719 (548h25m46.105129764s) [noise=true , address=145.79.227.52:4447 , piece_id=VJE7BFP3AFKNXNU54B5DJLZR7IGBJT77Y6XJ6Z2BVR2EVF2IXODQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.235369318 (548h25m46.106452353s) [noise=true , address=145.79.227.41:4086 , piece_id=UGAKRIVEDVOJFO3ISUOBOBBFL4YG355AE6MMPJT5HMRO7VV2XCFQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.236353903 (548h25m46.107436938s) [noise=true , address=145.79.227.42:4121 , piece_id=TAGBXIO45ABXWBOUJX45WFNWL2CHYLLJ6U5AR5PH6YYDAKMMVBNQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.237817741 (548h25m46.108900776s) [noise=true , address=45.91.116.13:4155 , piece_id=XHTNBU7SEGUBVM5DO2CVD3OM3TC2GZVF3URSRMM4UXNDXA7YRZQQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.239096295 (548h25m46.11017934s) [noise=true , address=145.79.227.3:4151 , piece_id=6A4E64JX3DXKRF56E7JVOOD7D5S4VXDYX3TOKUZLWX2J3K3FFBBQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.241113486 (548h25m46.112196521s) [noise=true , address=145.79.227.29:4943 , piece_id=2D7ZFO2KZZYUEXBXLBFLQBACU5Z3ZD6PVWSAAJRLI2QKT5JHR7OQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.242540152 (548h25m46.113623197s) [noise=true , address=145.79.227.2:4113 , piece_id=FNGUYX2MQFXIXFOGJOOGKJ6SHVKIS6Y6DPG73U63NKLL5AXY3JMQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.243204507 (548h25m46.114287553s) [noise=true , address=145.79.227.51:4410 , piece_id=IQU7577IBBMVWHKZPASYIAYUCYKUEEKOIGUCEGSYMPSJZXPAFO2A, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.243790012 (548h25m46.114873047s) [noise=true , address=77.90.31.13:4629 , piece_id=XLNQCJ4ECZ6SUE3NMNXMRGCDTK3G3MH2O6AVCQ2PRZJEZMN2N4OQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.244633690 (548h25m46.115716725s) [noise=true , address=145.79.227.38:4979 , piece_id=CN23SYUYZDQQBAG4UVDNTFC6NG75KLH3I4UEBXH6FQMXCVWFD5AQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:46.245128271 (548h25m46.116211316s) [noise=true , address=77.90.31.12:4590 , piece_id=XA6QJYXR2UXVF6UD64JCN2RHKIBMIKK6EADWZOC7YPH7DWAW7SJQ, seg_pos={0 0}, upload=480885]: Uploading piece... Dec 28 11:54:47.965255990 (548h25m47.836339025s) [upload=480885]: Next returned nil segment Dec 28 11:54:47.965274385 (548h25m47.83635744s) [upload=480885]: Waiting for error group managing segments... Dec 28 11:54:48.242105035 (548h25m48.11318807s) [noise=true , address=145.79.227.38:4979 , piece_id=CN23SYUYZDQQBAG4UVDNTFC6NG75KLH3I4UEBXH6FQMXCVWFD5AQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.246524470 (548h25m48.117607556s) [noise=true , address=145.79.227.41:4086 , piece_id=UGAKRIVEDVOJFO3ISUOBOBBFL4YG355AE6MMPJT5HMRO7VV2XCFQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.248178459 (548h25m48.119261525s) [noise=true , address=145.79.227.3:4151 , piece_id=6A4E64JX3DXKRF56E7JVOOD7D5S4VXDYX3TOKUZLWX2J3K3FFBBQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.249574237 (548h25m48.120657282s) [noise=true , address=145.79.227.52:4447 , piece_id=VJE7BFP3AFKNXNU54B5DJLZR7IGBJT77Y6XJ6Z2BVR2EVF2IXODQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.251150058 (548h25m48.122233104s) [noise=true , address=145.79.227.51:4410 , piece_id=IQU7577IBBMVWHKZPASYIAYUCYKUEEKOIGUCEGSYMPSJZXPAFO2A, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.251182420 (548h25m48.122265465s) [noise=true , address=145.79.227.48:4302 , piece_id=APT5L7ZGKZK6EN3KQYTDMFJJWJPZOIEW6UHF3VL6K43VTBIEEXVQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.262064651 (548h25m48.133147786s) [noise=true , address=145.79.227.36:4517 , piece_id=N4XA2FIFHVSORQZWLTEI3O6RBD2JGG7WOMQWP7N6YIXKJJBYTCVA, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.266426567 (548h25m48.137509602s) [noise=true , address=145.79.227.42:4121 , piece_id=TAGBXIO45ABXWBOUJX45WFNWL2CHYLLJ6U5AR5PH6YYDAKMMVBNQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.267961860 (548h25m48.139044905s) [noise=true , address=145.79.227.2:4113 , piece_id=FNGUYX2MQFXIXFOGJOOGKJ6SHVKIS6Y6DPG73U63NKLL5AXY3JMQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.277540830 (548h25m48.148623885s) [noise=true , address=145.79.227.29:4943 , piece_id=2D7ZFO2KZZYUEXBXLBFLQBACU5Z3ZD6PVWSAAJRLI2QKT5JHR7OQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.546989343 (548h25m48.418072378s) [noise=true , address=77.90.31.12:4590 , piece_id=XA6QJYXR2UXVF6UD64JCN2RHKIBMIKK6EADWZOC7YPH7DWAW7SJQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.572422142 (548h25m48.443505187s) [noise=true , address=77.90.31.13:4629 , piece_id=XLNQCJ4ECZ6SUE3NMNXMRGCDTK3G3MH2O6AVCQ2PRZJEZMN2N4OQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.607553367 (548h25m48.478636402s) [noise=true , address=45.91.116.11:4120 , piece_id=YMROM3S4LAMKIFHXUHCVILQJVCCOTIVIKPCF5A6BZ4XQOWDM2X2A, seg_pos={0 0}, upload=480885]: Done uploading piece. err: <nil> Dec 28 11:54:48.607584736 (548h25m48.478667781s) [seg_pos={0 0}, upload=480885]: Segment reached optimal threshold of 13 pieces. Dec 28 11:54:48.607870611 (548h25m48.478953656s) [noise=true , address=45.91.116.13:4155 , piece_id=XHTNBU7SEGUBVM5DO2CVD3OM3TC2GZVF3URSRMM4UXNDXA7YRZQQ, seg_pos={0 0}, upload=480885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 11:54:48.607878526 (548h25m48.478961561s) [noise=true , address=45.91.116.13:4155 , piece_id=XHTNBU7SEGUBVM5DO2CVD3OM3TC2GZVF3URSRMM4UXNDXA7YRZQQ, seg_pos={0 0}, upload=480885]: Long tail cancellation due to optimal threshold Dec 28 11:54:48.608262047 (548h25m48.479345082s) [noise=true , address=45.91.116.4:4939 , piece_id=2LFWZM3YTSMC4EVQF54TFONZCOLGPI5XP6EYKB4IGFRLJN2NM2QA, seg_pos={0 0}, upload=480885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 11:54:48.608270603 (548h25m48.479353638s) [noise=true , address=45.91.116.4:4939 , piece_id=2LFWZM3YTSMC4EVQF54TFONZCOLGPI5XP6EYKB4IGFRLJN2NM2QA, seg_pos={0 0}, upload=480885]: Long tail cancellation due to optimal threshold Dec 28 11:54:48.608410058 (548h25m48.479493093s) [noise=true , address=45.91.116.7:4012 , piece_id=4SQMNLDA6Y775WYSFWSPHQB6TOZANLUSK4M4OWPDJ3YQM277ORJA, seg_pos={0 0}, upload=480885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 11:54:48.608416050 (548h25m48.479499085s) [noise=true , address=45.91.116.7:4012 , piece_id=4SQMNLDA6Y775WYSFWSPHQB6TOZANLUSK4M4OWPDJ3YQM277ORJA, seg_pos={0 0}, upload=480885]: Long tail cancellation due to optimal threshold Dec 28 11:54:48.608433072 (548h25m48.479516118s) [seg_pos={0 0}, upload=480885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:54:48.608537421 (548h25m48.479620496s) [upload=480885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:54:48.608549293 (548h25m48.479632338s) [upload=480885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:54:48.899117843 (548h25m48.770200878s) [upload=480885]: Done uploading segments... Dec 28 11:54:48.899140557 (548h25m48.770223602s) [upload=480885]: Upload finished. err: <nil> Duration: 30.003797928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X........X...........X..X..............................] duration: 15.485074112s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 15.485101885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXX] duration: 5.001170326s trace: https://jaeger.dev.storj.io/trace/1ab807c9da77abef FAIL (2025-12-28 11:54:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001191337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X.......................XX....X.....XX...XXX.......X..] duration: 5.659715708s trace: https://jaeger.dev.storj.io/trace/385aebec7808a2cc FAIL (2025-12-28 11:54:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA) context deadline exceeded ======LOGS====== g segments... Dec 28 11:53:14.858101416 (548h24m14.729184451s) [upload=480865]: Got next segment. Inline: false Dec 28 11:53:14.858145991 (548h24m14.729229026s) [upload=480865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:53:14.858151711 (548h24m14.729234757s) [upload=480865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:53:15.107067968 (548h24m14.978151004s) [seg_pos={0 0}, upload=480865]: Begin upload segment... Dec 28 11:53:15.107127192 (548h24m14.978210227s) [seg_pos={0 0}, upload=480865]: Done begin upload segment. Dec 28 11:53:15.107155445 (548h24m14.97823848s) [noise=true , address=45.91.116.13:4156 , piece_id=TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.107200230 (548h24m14.978283266s) [noise=true , address=145.79.227.42:4123 , piece_id=EOMF37FKG4HG7PF7OK2BZFADVQGBYNRP4PJAFH5FTEJKIIMIZMDA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108114611 (548h24m14.979197656s) [noise=true , address=145.79.227.45:4229 , piece_id=RIGGRLAOFMQWMYPAPUHCFX4N2K2JCZMD5A6SVI23HDMI6RLZSRTQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108489826 (548h24m14.979572861s) [noise=true , address=145.79.227.46:4602 , piece_id=SRYBTVDXO675B46Q5JVSXEHWGDN5JZYSVEWCFUIA7RQC2MMDF5EA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108781261 (548h24m14.979864296s) [noise=true , address=145.79.227.50:4374 , piece_id=DOABKZAJEQS4YHVU2D3MUEW4I2KFPLVTAELFAFM4YSCM56LFQM7A, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109029403 (548h24m14.980112438s) [noise=true , address=145.79.227.12:4767 , piece_id=A62QMCXGZQKKVU2536WQ5BQMZOOB4MHCPYCG6J3E7VJRMOD6GINQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109310568 (548h24m14.980393613s) [noise=true , address=77.90.31.17:4770 , piece_id=TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109571476 (548h24m14.980654511s) [noise=true , address=145.79.227.27:4691 , piece_id=RVCF7WWEW7CJH463NF4A3Z5XQFDTIISBWL4IHQA2TQZL7XL2S2YA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.110043263 (548h24m14.981126299s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112190312 (548h24m14.983273347s) [noise=true , address=45.91.116.12:4225 , piece_id=OGG7APLVWXZTOY42TE335RAZ4STZ7SIRGVWXQPNVCBF5S5DLLUUQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112547242 (548h24m14.983630277s) [noise=true , address=145.79.227.11:4730 , piece_id=MYGIVC3DDLGOS426MDAI23YUF5SBHXMRCTQ7HKZDKOKS5OAGQ2OQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112865768 (548h24m14.983948803s) [noise=true , address=145.79.227.8:4330 , piece_id=5NAYAZOETI534T3LE7Z4GFGWIF22IRBZYLVB6BAOK3EHREZZ7VWQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113186529 (548h24m14.984269565s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113555421 (548h24m14.984638456s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113981443 (548h24m14.985064478s) [noise=true , address=145.79.227.15:4876 , piece_id=MUAXKWY7R4DZDHVLL5YC7URVFYXPL4E7NQDQBIBKLKX6USN6WNVA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.114365925 (548h24m14.98544896s) [noise=true , address=145.79.227.6:4257 , piece_id=3UJ3ILIV5VDH3RPTKDDEHYEZQF2MOIMQKTRJIQCG7RLUEDNTDJZA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:16.824716055 (548h24m16.695799101s) [upload=480865]: Next returned nil segment Dec 28 11:53:16.824771772 (548h24m16.695854827s) [upload=480865]: Waiting for error group managing segments... Dec 28 11:53:17.101067573 (548h24m16.972150618s) [noise=true , address=145.79.227.42:4123 , piece_id=EOMF37FKG4HG7PF7OK2BZFADVQGBYNRP4PJAFH5FTEJKIIMIZMDA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.114642191 (548h24m16.985725236s) [noise=true , address=145.79.227.8:4330 , piece_id=5NAYAZOETI534T3LE7Z4GFGWIF22IRBZYLVB6BAOK3EHREZZ7VWQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.115508311 (548h24m16.986591346s) [noise=true , address=145.79.227.27:4691 , piece_id=RVCF7WWEW7CJH463NF4A3Z5XQFDTIISBWL4IHQA2TQZL7XL2S2YA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.115947447 (548h24m16.987030492s) [noise=true , address=145.79.227.12:4767 , piece_id=A62QMCXGZQKKVU2536WQ5BQMZOOB4MHCPYCG6J3E7VJRMOD6GINQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.120196989 (548h24m16.991280034s) [noise=true , address=145.79.227.46:4602 , piece_id=SRYBTVDXO675B46Q5JVSXEHWGDN5JZYSVEWCFUIA7RQC2MMDF5EA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.121480303 (548h24m16.992563338s) [noise=true , address=145.79.227.6:4257 , piece_id=3UJ3ILIV5VDH3RPTKDDEHYEZQF2MOIMQKTRJIQCG7RLUEDNTDJZA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.121988009 (548h24m16.993071054s) [noise=true , address=145.79.227.15:4876 , piece_id=MUAXKWY7R4DZDHVLL5YC7URVFYXPL4E7NQDQBIBKLKX6USN6WNVA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.133233041 (548h24m17.004316086s) [noise=true , address=145.79.227.11:4730 , piece_id=MYGIVC3DDLGOS426MDAI23YUF5SBHXMRCTQ7HKZDKOKS5OAGQ2OQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.138766378 (548h24m17.009849423s) [noise=true , address=145.79.227.50:4374 , piece_id=DOABKZAJEQS4YHVU2D3MUEW4I2KFPLVTAELFAFM4YSCM56LFQM7A, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.145168769 (548h24m17.016251804s) [noise=true , address=145.79.227.45:4229 , piece_id=RIGGRLAOFMQWMYPAPUHCFX4N2K2JCZMD5A6SVI23HDMI6RLZSRTQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.318760005 (548h24m17.18984305s) [noise=true , address=77.90.31.17:4770 , piece_id=TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.352005038 (548h24m17.223088074s) [noise=true , address=45.91.116.12:4225 , piece_id=OGG7APLVWXZTOY42TE335RAZ4STZ7SIRGVWXQPNVCBF5S5DLLUUQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.406100555 (548h24m17.2771836s) [noise=true , address=45.91.116.13:4156 , piece_id=TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.406118439 (548h24m17.277201474s) [seg_pos={0 0}, upload=480865]: Segment reached optimal threshold of 13 pieces. Dec 28 11:53:17.406487401 (548h24m17.277570446s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection Dec 28 11:53:17.406495087 (548h24m17.277578122s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406636947 (548h24m17.277719982s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection Dec 28 11:53:17.406645563 (548h24m17.277728598s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406925346 (548h24m17.278008411s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 11:53:17.406942529 (548h24m17.278025574s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406975651 (548h24m17.278058696s) [seg_pos={0 0}, upload=480865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:53:17.407107151 (548h24m17.278190197s) [upload=480865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:53:17.407114505 (548h24m17.27819754s) [upload=480865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:53:17.698399890 (548h24m17.569482925s) [upload=480865]: Done uploading segments... Dec 28 11:53:17.698420479 (548h24m17.569503514s) [upload=480865]: Upload finished. err: <nil> Duration: 5.659745835s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X.......................XX....X.....XX...XXX.......X.] duration: 6.188865465s trace: https://jaeger.dev.storj.io/trace/385aebec7808a2cc FAIL (2025-12-28 11:54:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA) context deadline exceeded ======LOGS====== g segments... Dec 28 11:53:14.858101416 (548h24m14.729184451s) [upload=480865]: Got next segment. Inline: false Dec 28 11:53:14.858145991 (548h24m14.729229026s) [upload=480865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:53:14.858151711 (548h24m14.729234757s) [upload=480865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:53:15.107067968 (548h24m14.978151004s) [seg_pos={0 0}, upload=480865]: Begin upload segment... Dec 28 11:53:15.107127192 (548h24m14.978210227s) [seg_pos={0 0}, upload=480865]: Done begin upload segment. Dec 28 11:53:15.107155445 (548h24m14.97823848s) [noise=true , address=45.91.116.13:4156 , piece_id=TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.107200230 (548h24m14.978283266s) [noise=true , address=145.79.227.42:4123 , piece_id=EOMF37FKG4HG7PF7OK2BZFADVQGBYNRP4PJAFH5FTEJKIIMIZMDA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108114611 (548h24m14.979197656s) [noise=true , address=145.79.227.45:4229 , piece_id=RIGGRLAOFMQWMYPAPUHCFX4N2K2JCZMD5A6SVI23HDMI6RLZSRTQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108489826 (548h24m14.979572861s) [noise=true , address=145.79.227.46:4602 , piece_id=SRYBTVDXO675B46Q5JVSXEHWGDN5JZYSVEWCFUIA7RQC2MMDF5EA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108781261 (548h24m14.979864296s) [noise=true , address=145.79.227.50:4374 , piece_id=DOABKZAJEQS4YHVU2D3MUEW4I2KFPLVTAELFAFM4YSCM56LFQM7A, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109029403 (548h24m14.980112438s) [noise=true , address=145.79.227.12:4767 , piece_id=A62QMCXGZQKKVU2536WQ5BQMZOOB4MHCPYCG6J3E7VJRMOD6GINQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109310568 (548h24m14.980393613s) [noise=true , address=77.90.31.17:4770 , piece_id=TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109571476 (548h24m14.980654511s) [noise=true , address=145.79.227.27:4691 , piece_id=RVCF7WWEW7CJH463NF4A3Z5XQFDTIISBWL4IHQA2TQZL7XL2S2YA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.110043263 (548h24m14.981126299s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112190312 (548h24m14.983273347s) [noise=true , address=45.91.116.12:4225 , piece_id=OGG7APLVWXZTOY42TE335RAZ4STZ7SIRGVWXQPNVCBF5S5DLLUUQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112547242 (548h24m14.983630277s) [noise=true , address=145.79.227.11:4730 , piece_id=MYGIVC3DDLGOS426MDAI23YUF5SBHXMRCTQ7HKZDKOKS5OAGQ2OQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112865768 (548h24m14.983948803s) [noise=true , address=145.79.227.8:4330 , piece_id=5NAYAZOETI534T3LE7Z4GFGWIF22IRBZYLVB6BAOK3EHREZZ7VWQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113186529 (548h24m14.984269565s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113555421 (548h24m14.984638456s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113981443 (548h24m14.985064478s) [noise=true , address=145.79.227.15:4876 , piece_id=MUAXKWY7R4DZDHVLL5YC7URVFYXPL4E7NQDQBIBKLKX6USN6WNVA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.114365925 (548h24m14.98544896s) [noise=true , address=145.79.227.6:4257 , piece_id=3UJ3ILIV5VDH3RPTKDDEHYEZQF2MOIMQKTRJIQCG7RLUEDNTDJZA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:16.824716055 (548h24m16.695799101s) [upload=480865]: Next returned nil segment Dec 28 11:53:16.824771772 (548h24m16.695854827s) [upload=480865]: Waiting for error group managing segments... Dec 28 11:53:17.101067573 (548h24m16.972150618s) [noise=true , address=145.79.227.42:4123 , piece_id=EOMF37FKG4HG7PF7OK2BZFADVQGBYNRP4PJAFH5FTEJKIIMIZMDA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.114642191 (548h24m16.985725236s) [noise=true , address=145.79.227.8:4330 , piece_id=5NAYAZOETI534T3LE7Z4GFGWIF22IRBZYLVB6BAOK3EHREZZ7VWQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.115508311 (548h24m16.986591346s) [noise=true , address=145.79.227.27:4691 , piece_id=RVCF7WWEW7CJH463NF4A3Z5XQFDTIISBWL4IHQA2TQZL7XL2S2YA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.115947447 (548h24m16.987030492s) [noise=true , address=145.79.227.12:4767 , piece_id=A62QMCXGZQKKVU2536WQ5BQMZOOB4MHCPYCG6J3E7VJRMOD6GINQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.120196989 (548h24m16.991280034s) [noise=true , address=145.79.227.46:4602 , piece_id=SRYBTVDXO675B46Q5JVSXEHWGDN5JZYSVEWCFUIA7RQC2MMDF5EA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.121480303 (548h24m16.992563338s) [noise=true , address=145.79.227.6:4257 , piece_id=3UJ3ILIV5VDH3RPTKDDEHYEZQF2MOIMQKTRJIQCG7RLUEDNTDJZA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.121988009 (548h24m16.993071054s) [noise=true , address=145.79.227.15:4876 , piece_id=MUAXKWY7R4DZDHVLL5YC7URVFYXPL4E7NQDQBIBKLKX6USN6WNVA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.133233041 (548h24m17.004316086s) [noise=true , address=145.79.227.11:4730 , piece_id=MYGIVC3DDLGOS426MDAI23YUF5SBHXMRCTQ7HKZDKOKS5OAGQ2OQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.138766378 (548h24m17.009849423s) [noise=true , address=145.79.227.50:4374 , piece_id=DOABKZAJEQS4YHVU2D3MUEW4I2KFPLVTAELFAFM4YSCM56LFQM7A, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.145168769 (548h24m17.016251804s) [noise=true , address=145.79.227.45:4229 , piece_id=RIGGRLAOFMQWMYPAPUHCFX4N2K2JCZMD5A6SVI23HDMI6RLZSRTQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.318760005 (548h24m17.18984305s) [noise=true , address=77.90.31.17:4770 , piece_id=TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.352005038 (548h24m17.223088074s) [noise=true , address=45.91.116.12:4225 , piece_id=OGG7APLVWXZTOY42TE335RAZ4STZ7SIRGVWXQPNVCBF5S5DLLUUQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.406100555 (548h24m17.2771836s) [noise=true , address=45.91.116.13:4156 , piece_id=TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.406118439 (548h24m17.277201474s) [seg_pos={0 0}, upload=480865]: Segment reached optimal threshold of 13 pieces. Dec 28 11:53:17.406487401 (548h24m17.277570446s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection Dec 28 11:53:17.406495087 (548h24m17.277578122s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406636947 (548h24m17.277719982s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection Dec 28 11:53:17.406645563 (548h24m17.277728598s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406925346 (548h24m17.278008411s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 11:53:17.406942529 (548h24m17.278025574s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406975651 (548h24m17.278058696s) [seg_pos={0 0}, upload=480865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:53:17.407107151 (548h24m17.278190197s) [upload=480865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:53:17.407114505 (548h24m17.27819754s) [upload=480865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:53:17.698399890 (548h24m17.569482925s) [upload=480865]: Done uploading segments... Dec 28 11:53:17.698420479 (548h24m17.569503514s) [upload=480865]: Upload finished. err: <nil> Duration: 6.188901664s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXX] duration: 5.001760496s trace: https://jaeger.dev.storj.io/trace/2cd1c6dbbc50a57d FAIL (2025-12-28 11:54:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001786596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X........X...........X..X.............................] duration: 15.377959003s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:53:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 15.377988139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.......X.......................XX....X.....XX...XXX.......X] duration: 30.002348542s trace: https://jaeger.dev.storj.io/trace/385aebec7808a2cc FAIL (2025-12-28 11:53:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA) context deadline exceeded ======LOGS====== g segments... Dec 28 11:53:14.858101416 (548h24m14.729184451s) [upload=480865]: Got next segment. Inline: false Dec 28 11:53:14.858145991 (548h24m14.729229026s) [upload=480865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:53:14.858151711 (548h24m14.729234757s) [upload=480865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:53:15.107067968 (548h24m14.978151004s) [seg_pos={0 0}, upload=480865]: Begin upload segment... Dec 28 11:53:15.107127192 (548h24m14.978210227s) [seg_pos={0 0}, upload=480865]: Done begin upload segment. Dec 28 11:53:15.107155445 (548h24m14.97823848s) [noise=true , address=45.91.116.13:4156 , piece_id=TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.107200230 (548h24m14.978283266s) [noise=true , address=145.79.227.42:4123 , piece_id=EOMF37FKG4HG7PF7OK2BZFADVQGBYNRP4PJAFH5FTEJKIIMIZMDA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108114611 (548h24m14.979197656s) [noise=true , address=145.79.227.45:4229 , piece_id=RIGGRLAOFMQWMYPAPUHCFX4N2K2JCZMD5A6SVI23HDMI6RLZSRTQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108489826 (548h24m14.979572861s) [noise=true , address=145.79.227.46:4602 , piece_id=SRYBTVDXO675B46Q5JVSXEHWGDN5JZYSVEWCFUIA7RQC2MMDF5EA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.108781261 (548h24m14.979864296s) [noise=true , address=145.79.227.50:4374 , piece_id=DOABKZAJEQS4YHVU2D3MUEW4I2KFPLVTAELFAFM4YSCM56LFQM7A, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109029403 (548h24m14.980112438s) [noise=true , address=145.79.227.12:4767 , piece_id=A62QMCXGZQKKVU2536WQ5BQMZOOB4MHCPYCG6J3E7VJRMOD6GINQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109310568 (548h24m14.980393613s) [noise=true , address=77.90.31.17:4770 , piece_id=TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.109571476 (548h24m14.980654511s) [noise=true , address=145.79.227.27:4691 , piece_id=RVCF7WWEW7CJH463NF4A3Z5XQFDTIISBWL4IHQA2TQZL7XL2S2YA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.110043263 (548h24m14.981126299s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112190312 (548h24m14.983273347s) [noise=true , address=45.91.116.12:4225 , piece_id=OGG7APLVWXZTOY42TE335RAZ4STZ7SIRGVWXQPNVCBF5S5DLLUUQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112547242 (548h24m14.983630277s) [noise=true , address=145.79.227.11:4730 , piece_id=MYGIVC3DDLGOS426MDAI23YUF5SBHXMRCTQ7HKZDKOKS5OAGQ2OQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.112865768 (548h24m14.983948803s) [noise=true , address=145.79.227.8:4330 , piece_id=5NAYAZOETI534T3LE7Z4GFGWIF22IRBZYLVB6BAOK3EHREZZ7VWQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113186529 (548h24m14.984269565s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113555421 (548h24m14.984638456s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.113981443 (548h24m14.985064478s) [noise=true , address=145.79.227.15:4876 , piece_id=MUAXKWY7R4DZDHVLL5YC7URVFYXPL4E7NQDQBIBKLKX6USN6WNVA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:15.114365925 (548h24m14.98544896s) [noise=true , address=145.79.227.6:4257 , piece_id=3UJ3ILIV5VDH3RPTKDDEHYEZQF2MOIMQKTRJIQCG7RLUEDNTDJZA, seg_pos={0 0}, upload=480865]: Uploading piece... Dec 28 11:53:16.824716055 (548h24m16.695799101s) [upload=480865]: Next returned nil segment Dec 28 11:53:16.824771772 (548h24m16.695854827s) [upload=480865]: Waiting for error group managing segments... Dec 28 11:53:17.101067573 (548h24m16.972150618s) [noise=true , address=145.79.227.42:4123 , piece_id=EOMF37FKG4HG7PF7OK2BZFADVQGBYNRP4PJAFH5FTEJKIIMIZMDA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.114642191 (548h24m16.985725236s) [noise=true , address=145.79.227.8:4330 , piece_id=5NAYAZOETI534T3LE7Z4GFGWIF22IRBZYLVB6BAOK3EHREZZ7VWQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.115508311 (548h24m16.986591346s) [noise=true , address=145.79.227.27:4691 , piece_id=RVCF7WWEW7CJH463NF4A3Z5XQFDTIISBWL4IHQA2TQZL7XL2S2YA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.115947447 (548h24m16.987030492s) [noise=true , address=145.79.227.12:4767 , piece_id=A62QMCXGZQKKVU2536WQ5BQMZOOB4MHCPYCG6J3E7VJRMOD6GINQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.120196989 (548h24m16.991280034s) [noise=true , address=145.79.227.46:4602 , piece_id=SRYBTVDXO675B46Q5JVSXEHWGDN5JZYSVEWCFUIA7RQC2MMDF5EA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.121480303 (548h24m16.992563338s) [noise=true , address=145.79.227.6:4257 , piece_id=3UJ3ILIV5VDH3RPTKDDEHYEZQF2MOIMQKTRJIQCG7RLUEDNTDJZA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.121988009 (548h24m16.993071054s) [noise=true , address=145.79.227.15:4876 , piece_id=MUAXKWY7R4DZDHVLL5YC7URVFYXPL4E7NQDQBIBKLKX6USN6WNVA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.133233041 (548h24m17.004316086s) [noise=true , address=145.79.227.11:4730 , piece_id=MYGIVC3DDLGOS426MDAI23YUF5SBHXMRCTQ7HKZDKOKS5OAGQ2OQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.138766378 (548h24m17.009849423s) [noise=true , address=145.79.227.50:4374 , piece_id=DOABKZAJEQS4YHVU2D3MUEW4I2KFPLVTAELFAFM4YSCM56LFQM7A, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.145168769 (548h24m17.016251804s) [noise=true , address=145.79.227.45:4229 , piece_id=RIGGRLAOFMQWMYPAPUHCFX4N2K2JCZMD5A6SVI23HDMI6RLZSRTQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.318760005 (548h24m17.18984305s) [noise=true , address=77.90.31.17:4770 , piece_id=TX65LWQLZKJ5J5PHFWOV44EQTKNRRDX5ANG3D37JVHG7KPVXHOIQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.352005038 (548h24m17.223088074s) [noise=true , address=45.91.116.12:4225 , piece_id=OGG7APLVWXZTOY42TE335RAZ4STZ7SIRGVWXQPNVCBF5S5DLLUUQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.406100555 (548h24m17.2771836s) [noise=true , address=45.91.116.13:4156 , piece_id=TDO4R2DALSHVBGQIR3K4WAQHLZ4EPAPS3SOMXTTHC5M5C7EAZ3EA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: <nil> Dec 28 11:53:17.406118439 (548h24m17.277201474s) [seg_pos={0 0}, upload=480865]: Segment reached optimal threshold of 13 pieces. Dec 28 11:53:17.406487401 (548h24m17.277570446s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.8:4045: use of closed network connection Dec 28 11:53:17.406495087 (548h24m17.277578122s) [noise=true , address=45.91.116.8:4045 , piece_id=T763HU5PLVACEJ7HSXSZNBCTAZDIEA4UIPJZGJLJCLW4TA7IBWGQ, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406636947 (548h24m17.277719982s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56160->77.90.31.10:4519: use of closed network connection Dec 28 11:53:17.406645563 (548h24m17.277728598s) [noise=true , address=77.90.31.10:4519 , piece_id=KW3BWQX7QXILF54EI2LNTNQCEFP5DPB5HNEZST4AMAPQDDDHNRPA, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406925346 (548h24m17.278008411s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 11:53:17.406942529 (548h24m17.278025574s) [noise=true , address=45.91.116.5:4299 , piece_id=LABFPZA2QPO6Z6XHFWVMDJSE7K7O5777DDUFECSBS4IKJ56DKUFQ, seg_pos={0 0}, upload=480865]: Long tail cancellation due to optimal threshold Dec 28 11:53:17.406975651 (548h24m17.278058696s) [seg_pos={0 0}, upload=480865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:53:17.407107151 (548h24m17.278190197s) [upload=480865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:53:17.407114505 (548h24m17.27819754s) [upload=480865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:53:17.698399890 (548h24m17.569482925s) [upload=480865]: Done uploading segments... Dec 28 11:53:17.698420479 (548h24m17.569503514s) [upload=480865]: Upload finished. err: <nil> Duration: 30.002377707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X........X...........X..X............................] duration: 47.942830708s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:53:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 47.942858862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XXX] duration: 5.001342907s trace: https://jaeger.dev.storj.io/trace/32b8c3280235c21f FAIL (2025-12-28 11:53:27 UTC) orbiter warning: context deadline exceeded Duration: 5.001367304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X........X...........X..X...........................] duration: 33.432093507s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:52:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 33.432108865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.......X.......................XX....X.....XX...XXX.......] duration: 6.013420756s trace: https://jaeger.dev.storj.io/trace/659477683d696568 FAIL (2025-12-28 11:52:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q) context deadline exceeded ======LOGS====== Dec 28 11:46:50.596722197 (548h17m50.467805232s) [upload=480773]: Starting upload Dec 28 11:46:50.596785147 (548h17m50.467868183s) [upload=480773]: Done starting upload Dec 28 11:46:50.596945943 (548h17m50.468028988s) [upload=480773]: Uploading segments... Dec 28 11:46:50.597044300 (548h17m50.468127336s) [upload=480773]: Got next segment. Inline: false Dec 28 11:46:50.597090498 (548h17m50.468173533s) [upload=480773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:46:50.597097252 (548h17m50.468180297s) [upload=480773]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:46:50.842523763 (548h17m50.713606808s) [seg_pos={0 0}, upload=480773]: Begin upload segment... Dec 28 11:46:50.842577756 (548h17m50.713660811s) [seg_pos={0 0}, upload=480773]: Done begin upload segment. Dec 28 11:46:50.842619626 (548h17m50.713702661s) [noise=true , address=145.79.227.51:4410 , piece_id=J6J6UJX4IQQSI7I2GVEC545QM2N6H4SA3CSK6BO3OMB4QMMXEYAQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.842717963 (548h17m50.713801008s) [noise=true , address=145.79.227.14:4839 , piece_id=HXRQKFVVHDALI32SNNF4QZTLWKXOPHJDUKGYBXICIBCKVI6KUPKA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.843224687 (548h17m50.714307733s) [noise=true , address=145.79.227.11:4731 , piece_id=CQD42RUXSNLQ6JYVAJB4HWLEACAG3B3WLIFXI5LS3CY6WIEREMMQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.843679794 (548h17m50.714762849s) [noise=true , address=145.79.227.46:4603 , piece_id=FFJSD3HSVICHVI7V5WJM556XB467JLJIRFBIEIWXRAQBJ5HZH6OQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.844119522 (548h17m50.715202567s) [noise=true , address=77.90.31.13:4626 , piece_id=27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.844512700 (548h17m50.715595745s) [noise=true , address=145.79.227.47:4266 , piece_id=HPLDUZKXETDTN4Z4NGGKQK4WA4JZYYPNJ6SZHQHVLJRUQTR57QNQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.845063108 (548h17m50.716146143s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.845526731 (548h17m50.716609776s) [noise=true , address=145.79.227.50:4373 , piece_id=S6EJBMYLXDLB4GGTGSC2ADZSMYG73QGO4IVQWEW7FCX2A6ZINQNA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846167531 (548h17m50.717250566s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846188481 (548h17m50.717271516s) [noise=true , address=45.91.116.11:4118 , piece_id=BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846769698 (548h17m50.717852733s) [noise=true , address=145.79.227.39:4015 , piece_id=7T4WEVGEQPHMALPSWOCDYT2AVM73PVGX446EJLOV7ZB2IPJIXJ5Q, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.847176802 (548h17m50.718259838s) [noise=true , address=145.79.227.52:4445 , piece_id=UM5QVDFZZ6YUIEWLM7NSOF5L32ZM2BIGGXFQMXIUDMH3QP7OA2QA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.847670734 (548h17m50.718753779s) [noise=true , address=145.79.227.42:4121 , piece_id=RHVKJ43T4DSMM7ZSGEMZH5IKQJO2BIAYHJ6BD5ZIDFKBCR4IEQJQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.848209459 (548h17m50.719292494s) [noise=true , address=145.79.227.17:4972 , piece_id=UVNAIRYVT7ZRF4WHXUI63NZHZVDDUPW255A7IE4A777TI6EX3ITA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.848566930 (548h17m50.719649975s) [noise=true , address=77.90.31.19:4842 , piece_id=3MSF5KJS7QTSDUGVLHNA2BMI2JUKNS3E56SJBK27U6KLWSKA5PEQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.849049198 (548h17m50.720132233s) [noise=true , address=145.79.227.26:4655 , piece_id=6KZUZFJ3IUKO53NT4TKZBYJPEILSQLSM5OQEQE77TUTVJP32IANQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:52.467548681 (548h17m52.338631726s) [upload=480773]: Next returned nil segment Dec 28 11:46:52.467583578 (548h17m52.338666633s) [upload=480773]: Waiting for error group managing segments... Dec 28 11:46:52.854096120 (548h17m52.725179165s) [noise=true , address=145.79.227.39:4015 , piece_id=7T4WEVGEQPHMALPSWOCDYT2AVM73PVGX446EJLOV7ZB2IPJIXJ5Q, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.856871475 (548h17m52.72795451s) [noise=true , address=145.79.227.46:4603 , piece_id=FFJSD3HSVICHVI7V5WJM556XB467JLJIRFBIEIWXRAQBJ5HZH6OQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.857881879 (548h17m52.728964924s) [noise=true , address=145.79.227.17:4972 , piece_id=UVNAIRYVT7ZRF4WHXUI63NZHZVDDUPW255A7IE4A777TI6EX3ITA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.859295701 (548h17m52.730378746s) [noise=true , address=145.79.227.14:4839 , piece_id=HXRQKFVVHDALI32SNNF4QZTLWKXOPHJDUKGYBXICIBCKVI6KUPKA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.861026837 (548h17m52.732109882s) [noise=true , address=145.79.227.26:4655 , piece_id=6KZUZFJ3IUKO53NT4TKZBYJPEILSQLSM5OQEQE77TUTVJP32IANQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.863741557 (548h17m52.734824602s) [noise=true , address=145.79.227.50:4373 , piece_id=S6EJBMYLXDLB4GGTGSC2ADZSMYG73QGO4IVQWEW7FCX2A6ZINQNA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.864182076 (548h17m52.735265121s) [noise=true , address=145.79.227.47:4266 , piece_id=HPLDUZKXETDTN4Z4NGGKQK4WA4JZYYPNJ6SZHQHVLJRUQTR57QNQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.866334654 (548h17m52.737417699s) [noise=true , address=145.79.227.51:4410 , piece_id=J6J6UJX4IQQSI7I2GVEC545QM2N6H4SA3CSK6BO3OMB4QMMXEYAQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.867731293 (548h17m52.738814328s) [noise=true , address=145.79.227.42:4121 , piece_id=RHVKJ43T4DSMM7ZSGEMZH5IKQJO2BIAYHJ6BD5ZIDFKBCR4IEQJQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.871317001 (548h17m52.742400036s) [noise=true , address=145.79.227.52:4445 , piece_id=UM5QVDFZZ6YUIEWLM7NSOF5L32ZM2BIGGXFQMXIUDMH3QP7OA2QA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.882538299 (548h17m52.753621344s) [noise=true , address=145.79.227.11:4731 , piece_id=CQD42RUXSNLQ6JYVAJB4HWLEACAG3B3WLIFXI5LS3CY6WIEREMMQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.043334249 (548h17m52.914417294s) [noise=true , address=77.90.31.19:4842 , piece_id=3MSF5KJS7QTSDUGVLHNA2BMI2JUKNS3E56SJBK27U6KLWSKA5PEQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129100552 (548h17m53.000183597s) [noise=true , address=77.90.31.13:4626 , piece_id=27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129128024 (548h17m53.00021108s) [seg_pos={0 0}, upload=480773]: Segment reached optimal threshold of 13 pieces. Dec 28 11:46:53.129162600 (548h17m53.000245645s) [noise=true , address=45.91.116.11:4118 , piece_id=BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129330580 (548h17m53.000413625s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection Dec 28 11:46:53.129340870 (548h17m53.000423905s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Long tail cancellation due to optimal threshold Dec 28 11:46:53.129509952 (548h17m53.000592997s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 11:46:53.129519259 (548h17m53.000602294s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Long tail cancellation due to optimal threshold Dec 28 11:46:53.129540931 (548h17m53.000623976s) [seg_pos={0 0}, upload=480773]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 11:46:53.129620672 (548h17m53.000703718s) [upload=480773]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:46:53.129628888 (548h17m53.000711933s) [upload=480773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:46:53.421448279 (548h17m53.292531324s) [upload=480773]: Done uploading segments... Dec 28 11:46:53.421479930 (548h17m53.292562985s) [upload=480773]: Upload finished. err: <nil> Duration: 6.013446725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.......X.......................XX....X.....XX...XXX......] duration: 24.001825782s trace: https://jaeger.dev.storj.io/trace/659477683d696568 FAIL (2025-12-28 11:52:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q) context deadline exceeded ======LOGS====== Dec 28 11:46:50.596722197 (548h17m50.467805232s) [upload=480773]: Starting upload Dec 28 11:46:50.596785147 (548h17m50.467868183s) [upload=480773]: Done starting upload Dec 28 11:46:50.596945943 (548h17m50.468028988s) [upload=480773]: Uploading segments... Dec 28 11:46:50.597044300 (548h17m50.468127336s) [upload=480773]: Got next segment. Inline: false Dec 28 11:46:50.597090498 (548h17m50.468173533s) [upload=480773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:46:50.597097252 (548h17m50.468180297s) [upload=480773]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:46:50.842523763 (548h17m50.713606808s) [seg_pos={0 0}, upload=480773]: Begin upload segment... Dec 28 11:46:50.842577756 (548h17m50.713660811s) [seg_pos={0 0}, upload=480773]: Done begin upload segment. Dec 28 11:46:50.842619626 (548h17m50.713702661s) [noise=true , address=145.79.227.51:4410 , piece_id=J6J6UJX4IQQSI7I2GVEC545QM2N6H4SA3CSK6BO3OMB4QMMXEYAQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.842717963 (548h17m50.713801008s) [noise=true , address=145.79.227.14:4839 , piece_id=HXRQKFVVHDALI32SNNF4QZTLWKXOPHJDUKGYBXICIBCKVI6KUPKA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.843224687 (548h17m50.714307733s) [noise=true , address=145.79.227.11:4731 , piece_id=CQD42RUXSNLQ6JYVAJB4HWLEACAG3B3WLIFXI5LS3CY6WIEREMMQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.843679794 (548h17m50.714762849s) [noise=true , address=145.79.227.46:4603 , piece_id=FFJSD3HSVICHVI7V5WJM556XB467JLJIRFBIEIWXRAQBJ5HZH6OQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.844119522 (548h17m50.715202567s) [noise=true , address=77.90.31.13:4626 , piece_id=27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.844512700 (548h17m50.715595745s) [noise=true , address=145.79.227.47:4266 , piece_id=HPLDUZKXETDTN4Z4NGGKQK4WA4JZYYPNJ6SZHQHVLJRUQTR57QNQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.845063108 (548h17m50.716146143s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.845526731 (548h17m50.716609776s) [noise=true , address=145.79.227.50:4373 , piece_id=S6EJBMYLXDLB4GGTGSC2ADZSMYG73QGO4IVQWEW7FCX2A6ZINQNA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846167531 (548h17m50.717250566s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846188481 (548h17m50.717271516s) [noise=true , address=45.91.116.11:4118 , piece_id=BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846769698 (548h17m50.717852733s) [noise=true , address=145.79.227.39:4015 , piece_id=7T4WEVGEQPHMALPSWOCDYT2AVM73PVGX446EJLOV7ZB2IPJIXJ5Q, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.847176802 (548h17m50.718259838s) [noise=true , address=145.79.227.52:4445 , piece_id=UM5QVDFZZ6YUIEWLM7NSOF5L32ZM2BIGGXFQMXIUDMH3QP7OA2QA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.847670734 (548h17m50.718753779s) [noise=true , address=145.79.227.42:4121 , piece_id=RHVKJ43T4DSMM7ZSGEMZH5IKQJO2BIAYHJ6BD5ZIDFKBCR4IEQJQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.848209459 (548h17m50.719292494s) [noise=true , address=145.79.227.17:4972 , piece_id=UVNAIRYVT7ZRF4WHXUI63NZHZVDDUPW255A7IE4A777TI6EX3ITA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.848566930 (548h17m50.719649975s) [noise=true , address=77.90.31.19:4842 , piece_id=3MSF5KJS7QTSDUGVLHNA2BMI2JUKNS3E56SJBK27U6KLWSKA5PEQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.849049198 (548h17m50.720132233s) [noise=true , address=145.79.227.26:4655 , piece_id=6KZUZFJ3IUKO53NT4TKZBYJPEILSQLSM5OQEQE77TUTVJP32IANQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:52.467548681 (548h17m52.338631726s) [upload=480773]: Next returned nil segment Dec 28 11:46:52.467583578 (548h17m52.338666633s) [upload=480773]: Waiting for error group managing segments... Dec 28 11:46:52.854096120 (548h17m52.725179165s) [noise=true , address=145.79.227.39:4015 , piece_id=7T4WEVGEQPHMALPSWOCDYT2AVM73PVGX446EJLOV7ZB2IPJIXJ5Q, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.856871475 (548h17m52.72795451s) [noise=true , address=145.79.227.46:4603 , piece_id=FFJSD3HSVICHVI7V5WJM556XB467JLJIRFBIEIWXRAQBJ5HZH6OQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.857881879 (548h17m52.728964924s) [noise=true , address=145.79.227.17:4972 , piece_id=UVNAIRYVT7ZRF4WHXUI63NZHZVDDUPW255A7IE4A777TI6EX3ITA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.859295701 (548h17m52.730378746s) [noise=true , address=145.79.227.14:4839 , piece_id=HXRQKFVVHDALI32SNNF4QZTLWKXOPHJDUKGYBXICIBCKVI6KUPKA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.861026837 (548h17m52.732109882s) [noise=true , address=145.79.227.26:4655 , piece_id=6KZUZFJ3IUKO53NT4TKZBYJPEILSQLSM5OQEQE77TUTVJP32IANQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.863741557 (548h17m52.734824602s) [noise=true , address=145.79.227.50:4373 , piece_id=S6EJBMYLXDLB4GGTGSC2ADZSMYG73QGO4IVQWEW7FCX2A6ZINQNA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.864182076 (548h17m52.735265121s) [noise=true , address=145.79.227.47:4266 , piece_id=HPLDUZKXETDTN4Z4NGGKQK4WA4JZYYPNJ6SZHQHVLJRUQTR57QNQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.866334654 (548h17m52.737417699s) [noise=true , address=145.79.227.51:4410 , piece_id=J6J6UJX4IQQSI7I2GVEC545QM2N6H4SA3CSK6BO3OMB4QMMXEYAQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.867731293 (548h17m52.738814328s) [noise=true , address=145.79.227.42:4121 , piece_id=RHVKJ43T4DSMM7ZSGEMZH5IKQJO2BIAYHJ6BD5ZIDFKBCR4IEQJQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.871317001 (548h17m52.742400036s) [noise=true , address=145.79.227.52:4445 , piece_id=UM5QVDFZZ6YUIEWLM7NSOF5L32ZM2BIGGXFQMXIUDMH3QP7OA2QA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.882538299 (548h17m52.753621344s) [noise=true , address=145.79.227.11:4731 , piece_id=CQD42RUXSNLQ6JYVAJB4HWLEACAG3B3WLIFXI5LS3CY6WIEREMMQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.043334249 (548h17m52.914417294s) [noise=true , address=77.90.31.19:4842 , piece_id=3MSF5KJS7QTSDUGVLHNA2BMI2JUKNS3E56SJBK27U6KLWSKA5PEQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129100552 (548h17m53.000183597s) [noise=true , address=77.90.31.13:4626 , piece_id=27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129128024 (548h17m53.00021108s) [seg_pos={0 0}, upload=480773]: Segment reached optimal threshold of 13 pieces. Dec 28 11:46:53.129162600 (548h17m53.000245645s) [noise=true , address=45.91.116.11:4118 , piece_id=BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129330580 (548h17m53.000413625s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection Dec 28 11:46:53.129340870 (548h17m53.000423905s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Long tail cancellation due to optimal threshold Dec 28 11:46:53.129509952 (548h17m53.000592997s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 11:46:53.129519259 (548h17m53.000602294s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Long tail cancellation due to optimal threshold Dec 28 11:46:53.129540931 (548h17m53.000623976s) [seg_pos={0 0}, upload=480773]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 11:46:53.129620672 (548h17m53.000703718s) [upload=480773]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:46:53.129628888 (548h17m53.000711933s) [upload=480773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:46:53.421448279 (548h17m53.292531324s) [upload=480773]: Done uploading segments... Dec 28 11:46:53.421479930 (548h17m53.292562985s) [upload=480773]: Upload finished. err: <nil> Duration: 24.001857423s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.XX] duration: 5.001236404s trace: https://jaeger.dev.storj.io/trace/69fb0cb55c299914 FAIL (2025-12-28 11:52:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001256963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X........X...........X..X..........................] duration: 41.795764492s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:52:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 41.795791345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.X] duration: 5.001687364s trace: https://jaeger.dev.storj.io/trace/2fc670cbb873574e FAIL (2025-12-28 11:51:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001712883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX.] duration: 3.524887411s trace: https://jaeger.dev.storj.io/trace/10a6c3ef840f638c FAIL (2025-12-28 11:51:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524940963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X........X...........X..X.........................] duration: 40.868935797s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:51:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 40.868958231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X.......X.......................XX....X.....XX...XXX.....] duration: 5.684097905s trace: https://jaeger.dev.storj.io/trace/659477683d696568 FAIL (2025-12-28 11:50:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q) context deadline exceeded ======LOGS====== Dec 28 11:46:50.596722197 (548h17m50.467805232s) [upload=480773]: Starting upload Dec 28 11:46:50.596785147 (548h17m50.467868183s) [upload=480773]: Done starting upload Dec 28 11:46:50.596945943 (548h17m50.468028988s) [upload=480773]: Uploading segments... Dec 28 11:46:50.597044300 (548h17m50.468127336s) [upload=480773]: Got next segment. Inline: false Dec 28 11:46:50.597090498 (548h17m50.468173533s) [upload=480773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:46:50.597097252 (548h17m50.468180297s) [upload=480773]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:46:50.842523763 (548h17m50.713606808s) [seg_pos={0 0}, upload=480773]: Begin upload segment... Dec 28 11:46:50.842577756 (548h17m50.713660811s) [seg_pos={0 0}, upload=480773]: Done begin upload segment. Dec 28 11:46:50.842619626 (548h17m50.713702661s) [noise=true , address=145.79.227.51:4410 , piece_id=J6J6UJX4IQQSI7I2GVEC545QM2N6H4SA3CSK6BO3OMB4QMMXEYAQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.842717963 (548h17m50.713801008s) [noise=true , address=145.79.227.14:4839 , piece_id=HXRQKFVVHDALI32SNNF4QZTLWKXOPHJDUKGYBXICIBCKVI6KUPKA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.843224687 (548h17m50.714307733s) [noise=true , address=145.79.227.11:4731 , piece_id=CQD42RUXSNLQ6JYVAJB4HWLEACAG3B3WLIFXI5LS3CY6WIEREMMQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.843679794 (548h17m50.714762849s) [noise=true , address=145.79.227.46:4603 , piece_id=FFJSD3HSVICHVI7V5WJM556XB467JLJIRFBIEIWXRAQBJ5HZH6OQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.844119522 (548h17m50.715202567s) [noise=true , address=77.90.31.13:4626 , piece_id=27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.844512700 (548h17m50.715595745s) [noise=true , address=145.79.227.47:4266 , piece_id=HPLDUZKXETDTN4Z4NGGKQK4WA4JZYYPNJ6SZHQHVLJRUQTR57QNQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.845063108 (548h17m50.716146143s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.845526731 (548h17m50.716609776s) [noise=true , address=145.79.227.50:4373 , piece_id=S6EJBMYLXDLB4GGTGSC2ADZSMYG73QGO4IVQWEW7FCX2A6ZINQNA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846167531 (548h17m50.717250566s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846188481 (548h17m50.717271516s) [noise=true , address=45.91.116.11:4118 , piece_id=BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846769698 (548h17m50.717852733s) [noise=true , address=145.79.227.39:4015 , piece_id=7T4WEVGEQPHMALPSWOCDYT2AVM73PVGX446EJLOV7ZB2IPJIXJ5Q, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.847176802 (548h17m50.718259838s) [noise=true , address=145.79.227.52:4445 , piece_id=UM5QVDFZZ6YUIEWLM7NSOF5L32ZM2BIGGXFQMXIUDMH3QP7OA2QA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.847670734 (548h17m50.718753779s) [noise=true , address=145.79.227.42:4121 , piece_id=RHVKJ43T4DSMM7ZSGEMZH5IKQJO2BIAYHJ6BD5ZIDFKBCR4IEQJQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.848209459 (548h17m50.719292494s) [noise=true , address=145.79.227.17:4972 , piece_id=UVNAIRYVT7ZRF4WHXUI63NZHZVDDUPW255A7IE4A777TI6EX3ITA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.848566930 (548h17m50.719649975s) [noise=true , address=77.90.31.19:4842 , piece_id=3MSF5KJS7QTSDUGVLHNA2BMI2JUKNS3E56SJBK27U6KLWSKA5PEQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.849049198 (548h17m50.720132233s) [noise=true , address=145.79.227.26:4655 , piece_id=6KZUZFJ3IUKO53NT4TKZBYJPEILSQLSM5OQEQE77TUTVJP32IANQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:52.467548681 (548h17m52.338631726s) [upload=480773]: Next returned nil segment Dec 28 11:46:52.467583578 (548h17m52.338666633s) [upload=480773]: Waiting for error group managing segments... Dec 28 11:46:52.854096120 (548h17m52.725179165s) [noise=true , address=145.79.227.39:4015 , piece_id=7T4WEVGEQPHMALPSWOCDYT2AVM73PVGX446EJLOV7ZB2IPJIXJ5Q, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.856871475 (548h17m52.72795451s) [noise=true , address=145.79.227.46:4603 , piece_id=FFJSD3HSVICHVI7V5WJM556XB467JLJIRFBIEIWXRAQBJ5HZH6OQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.857881879 (548h17m52.728964924s) [noise=true , address=145.79.227.17:4972 , piece_id=UVNAIRYVT7ZRF4WHXUI63NZHZVDDUPW255A7IE4A777TI6EX3ITA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.859295701 (548h17m52.730378746s) [noise=true , address=145.79.227.14:4839 , piece_id=HXRQKFVVHDALI32SNNF4QZTLWKXOPHJDUKGYBXICIBCKVI6KUPKA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.861026837 (548h17m52.732109882s) [noise=true , address=145.79.227.26:4655 , piece_id=6KZUZFJ3IUKO53NT4TKZBYJPEILSQLSM5OQEQE77TUTVJP32IANQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.863741557 (548h17m52.734824602s) [noise=true , address=145.79.227.50:4373 , piece_id=S6EJBMYLXDLB4GGTGSC2ADZSMYG73QGO4IVQWEW7FCX2A6ZINQNA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.864182076 (548h17m52.735265121s) [noise=true , address=145.79.227.47:4266 , piece_id=HPLDUZKXETDTN4Z4NGGKQK4WA4JZYYPNJ6SZHQHVLJRUQTR57QNQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.866334654 (548h17m52.737417699s) [noise=true , address=145.79.227.51:4410 , piece_id=J6J6UJX4IQQSI7I2GVEC545QM2N6H4SA3CSK6BO3OMB4QMMXEYAQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.867731293 (548h17m52.738814328s) [noise=true , address=145.79.227.42:4121 , piece_id=RHVKJ43T4DSMM7ZSGEMZH5IKQJO2BIAYHJ6BD5ZIDFKBCR4IEQJQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.871317001 (548h17m52.742400036s) [noise=true , address=145.79.227.52:4445 , piece_id=UM5QVDFZZ6YUIEWLM7NSOF5L32ZM2BIGGXFQMXIUDMH3QP7OA2QA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.882538299 (548h17m52.753621344s) [noise=true , address=145.79.227.11:4731 , piece_id=CQD42RUXSNLQ6JYVAJB4HWLEACAG3B3WLIFXI5LS3CY6WIEREMMQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.043334249 (548h17m52.914417294s) [noise=true , address=77.90.31.19:4842 , piece_id=3MSF5KJS7QTSDUGVLHNA2BMI2JUKNS3E56SJBK27U6KLWSKA5PEQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129100552 (548h17m53.000183597s) [noise=true , address=77.90.31.13:4626 , piece_id=27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129128024 (548h17m53.00021108s) [seg_pos={0 0}, upload=480773]: Segment reached optimal threshold of 13 pieces. Dec 28 11:46:53.129162600 (548h17m53.000245645s) [noise=true , address=45.91.116.11:4118 , piece_id=BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129330580 (548h17m53.000413625s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection Dec 28 11:46:53.129340870 (548h17m53.000423905s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Long tail cancellation due to optimal threshold Dec 28 11:46:53.129509952 (548h17m53.000592997s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 11:46:53.129519259 (548h17m53.000602294s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Long tail cancellation due to optimal threshold Dec 28 11:46:53.129540931 (548h17m53.000623976s) [seg_pos={0 0}, upload=480773]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 11:46:53.129620672 (548h17m53.000703718s) [upload=480773]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:46:53.129628888 (548h17m53.000711933s) [upload=480773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:46:53.421448279 (548h17m53.292531324s) [upload=480773]: Done uploading segments... Dec 28 11:46:53.421479930 (548h17m53.292562985s) [upload=480773]: Upload finished. err: <nil> Duration: 5.684123923s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....XX.XX] duration: 5.001074619s trace: https://jaeger.dev.storj.io/trace/10a6c3ef840f638c FAIL (2025-12-28 11:50:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001102251s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)