Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.42022564s trace: https://jaeger.dev.storj.io/trace/7173e01acd2fefef FAIL (2025-11-04 09:38:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420339237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.909643282s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:37:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.909670324s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.393650625s trace: https://jaeger.dev.storj.io/trace/7173e01acd2fefef FAIL (2025-11-04 09:37:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.393676274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.742802358s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:36:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.742836432s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.402938691s trace: https://jaeger.dev.storj.io/trace/7173e01acd2fefef FAIL (2025-11-04 09:36:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.403022991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.912291884s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:36:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.912320911s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 4.777789043s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:36:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.777816115s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 5.001126283s trace: https://jaeger.dev.storj.io/trace/7173e01acd2fefef FAIL (2025-11-04 09:35:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.502156034s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:35:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.502187265s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.915910938s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:35:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.915959722s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.397478156s trace: https://jaeger.dev.storj.io/trace/7173e01acd2fefef FAIL (2025-11-04 09:34:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.397515798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.418527932s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:34:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.418562599s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.400311189s trace: https://jaeger.dev.storj.io/trace/7173e01acd2fefef FAIL (2025-11-04 09:33:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.400336506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.452893017s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:33:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.452942031s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.391721492s trace: https://jaeger.dev.storj.io/trace/7173e01acd2fefef FAIL (2025-11-04 09:32:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39173597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.807764466s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:32:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.807788944s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 4.794953025s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:31:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.794981089s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001735895s trace: https://jaeger.dev.storj.io/trace/7173e01acd2fefef FAIL (2025-11-04 09:31:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001761935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.838206158s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:31:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.838238018s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.424357035s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:30:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.424385679s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.881650433s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:29:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.881667366s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.469968s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:29:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.469989071s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.407426499s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:28:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40745275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.40417081s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:28:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.404199696s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.779060181s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:28:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.779093174s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.383948481s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:28:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383979632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.411429098s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:27:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.411455199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.360835861s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:27:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.360875385s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.380205254s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:26:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.380238397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.90750888s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:26:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.907526074s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m46.283045405s trace: https://jaeger.dev.storj.io/trace/fe1b57ca38f9d5 FAIL (2025-11-04 09:26:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.833784389 (117h44m7.529633142s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.834635689 (117h44m7.530484443s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.835564679 (117h44m7.531413442s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.837103248 (117h44m7.532952001s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.838931159 (117h44m7.534779902s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.840976174 (117h44m7.536824927s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.843209847 (117h44m7.53905863s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.845480721 (117h44m7.541329474s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.847906691 (117h44m7.543755444s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:35.850674971 (117h44m7.546523724s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Uploading piece... Nov 4 09:24:36.664111590 (117h44m8.359960343s) [upload=103704]: Next returned nil segment Nov 4 09:24:36.664131899 (117h44m8.359980642s) [upload=103704]: Waiting for error group managing segments... Nov 4 09:24:37.045621097 (117h44m8.74146985s) [noise=true , address=81.171.14.70:4040 , piece_id=GEPJSR6OOFLMWVK6L6P5M6EUVQKWE22KPHNVIDHMADZKFJ5FUM4Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.056210781 (117h44m8.752059534s) [noise=true , address=81.171.14.70:4033 , piece_id=ETQB3UOTGZ43POUYWXGKG3E4ME7R6RSLXOEIXS3JNTGCNS6P7TQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.060042558 (117h44m8.755891301s) [noise=true , address=81.171.14.70:4028 , piece_id=YMF72OFENI7X7H3YMB2JQ6OIRLRTTEMEXEZAQDOTYQZL5Y4S75NA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.064340322 (117h44m8.760189075s) [noise=true , address=81.171.14.70:4031 , piece_id=PNZH3UOW5IGQSEY72TRKSREPCDQQEC6FORXAVHMWXFOCFY6A4RQQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.102924393 (117h44m8.798773146s) [noise=true , address=89.149.201.147:4019 , piece_id=NXJWZY4FFD6CRVDU3PYON75MTOD3BWJKNZOA4Y7OOJZFR3DIRD3Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.105278415 (117h44m8.801127158s) [noise=true , address=81.171.14.70:4045 , piece_id=Z2RWY6FAMDKG7JNN57NO3RQSTNDQDB7MCGFQJ425KOJIF7QREDEA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.115872909 (117h44m8.811721653s) [noise=true , address=89.149.201.147:4015 , piece_id=YJEYCADFGHOOF5U4RZ4OJI7CA5LXAB3GODZTT6N4DVO57YCKMZSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.126338458 (117h44m8.822187201s) [noise=true , address=81.171.14.70:4035 , piece_id=O7VUIYG42YBUKTLL32W7PAY6EWO5VOOUIVGWQBIDS2VANVZK4HIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.191443420 (117h44m8.887292173s) [noise=true , address=89.149.201.147:4003 , piece_id=5JOP42Z6EVY6RM33OIVFFZ6RQVWOARYAKYLFBAWX5DJLATWW624Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.194241959 (117h44m8.890090712s) [noise=true , address=89.149.201.147:4008 , piece_id=RSM2A63DSMYD6OX3FW2HOF3G7EKBM2YGB4G75QIWTUBACF3H3MFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.551317990 (117h44m9.247166743s) [noise=true , address=81.171.14.70:4026 , piece_id=AJLQVEFRMVKNFTZGY5OW644BENCT23HVOSQ7GPADORG7RXHGTSNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568367696 (117h44m9.264216449s) [noise=true , address=81.171.14.70:4030 , piece_id=KEDQVSOV6REFMOK6DJOQFQ4KW2SIIHHDNK6S52TGOBUXNH47RD5Q, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.568875453 (117h44m9.264724196s) [noise=true , address=81.171.14.70:4034 , piece_id=MQ4ICAT4UQQYMW7NM5OITWDWSG25ENMTUF3GJBFYD22FOZXWQSFA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.589097079 (117h44m9.284945832s) [noise=true , address=81.171.14.70:4047 , piece_id=XV4RTQOSITSYJSOBS6Y6VHAAPGJMBOFN2S7HYJ4PIGJAWJVLRL6A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.601272223 (117h44m9.297120976s) [noise=true , address=89.149.201.147:4023 , piece_id=Q7UEKYWHTRWNIPUZPQNRIWZA5XNM5XNPCYJ44PN6NY6IUN3F32IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.609350455 (117h44m9.305199198s) [noise=true , address=89.149.201.147:4020 , piece_id=WDXS33TX2MWWF5EVV6QGD6QEWUG26TB2USF5YHHFMU5LBBCTR3IA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.629258303 (117h44m9.325107056s) [noise=true , address=185.167.204.205:4026 , piece_id=6T6FV6V432RZSVC3LAJSBWBR5K3ZEFHC5P7PRU4QOIRRVSZX7MIA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.654483166 (117h44m9.350331909s) [noise=true , address=185.167.204.206:4038 , piece_id=FPRBRF7CXQXFQ3K4BRU3S5TJHW7QUQZ3AVWB2EVGG6T743DT6U3A, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.660526313 (117h44m9.356375057s) [noise=true , address=89.149.201.147:4017 , piece_id=DGOKI3WO2NFQSRWZO46ZINJJ4DQHYQ6A2KFDMJRFP2ZWAPT2TUDA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.663703333 (117h44m9.359552076s) [noise=true , address=185.167.204.206:4047 , piece_id=3EYIA3ZWROJHPB3AHCOBKIJOJ6T53MKHC4ENWXE4WMLKQKIRRRUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.695782357 (117h44m9.39163111s) [noise=true , address=185.167.204.206:4048 , piece_id=2CDL4Y6NQP4SNSUTDUND7SYMDYVZZGH6RD4ZBYHS3YKZFIYW5RUQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.731635860 (117h44m9.427484613s) [noise=true , address=185.167.204.205:4027 , piece_id=WN4VI6SD7ZKUUPJHY3HPR7NWT654MI2BJJFNSX5DPWDVA75FA5SA, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.746959419 (117h44m9.442808172s) [noise=true , address=185.167.204.205:4003 , piece_id=4PIRLQOHQLDY5XUZX4IBJWH77VFN4K6J34DEE2KQTPWGD4HMAHLQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.752846228 (117h44m9.448694982s) [noise=true , address=185.167.204.206:4041 , piece_id=VUECGW4KJIHVCPYMUZNFDV4MLEZXYTPCVS5WDFJT5ESGG36CDQYQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.758217416 (117h44m9.454066169s) [noise=true , address=185.167.204.205:4021 , piece_id=C3VLJX3FTH5FCGL5Y544RB3NIUJQABUJB6PGPZWF5FZHUNYBEWSQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805080161 (117h44m9.500928914s) [noise=true , address=185.167.204.206:4058 , piece_id=ZQWHADQTIBRCBEKWJJWNSVP25BTN4K24A5QOZQFICSDL6PD2JIFQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: <nil> Nov 4 09:24:37.805112914 (117h44m9.500961667s) [seg_pos={0 0}, upload=103704]: Segment reached optimal threshold of 26 pieces. Nov 4 09:24:37.805362599 (117h44m9.501211352s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805371898 (117h44m9.501220651s) [noise=true , address=185.167.204.206:4045 , piece_id=OVJKACGPZFX6X5MCQTE4HCIJ2YS42CWXMJXJGG6BGQANYEBS6YNQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805623827 (117h44m9.50147257s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF Nov 4 09:24:37.805676367 (117h44m9.50152512s) [noise=true , address=185.167.204.205:4022 , piece_id=HBXNICVQGEJD7IM6SKUF2BMRZX4NBX2PPE4P6GNHB3WP3RS2DHXQ, seg_pos={0 0}, upload=103704]: Long tail cancellation due to optimal threshold Nov 4 09:24:37.805719439 (117h44m9.501568182s) [seg_pos={0 0}, upload=103704]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 4 09:24:37.805868984 (117h44m9.501717737s) [upload=103704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 4 09:24:37.805875586 (117h44m9.501724329s) [upload=103704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 4 09:26:21.434315496 (117h45m53.130164239s) [upload=103704]: Done uploading segments... Nov 4 09:26:21.434377824 (117h45m53.130226567s) [upload=103704]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m46.283060143s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.397833522s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:26:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.397866405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.459729914s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:25:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459762076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.344507029s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:25:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.344535924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.358607018s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:24:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.358668224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.380963309s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:23:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.380991352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.350487606s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:22:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.35051665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.351120102s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:21:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.351146301s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.412997081s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:21:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.41303914s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.44081364s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:20:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440834419s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.394052763s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:20:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.3940824s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.451464674s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:19:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45149419s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.397144242s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:19:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.397162015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.412234404s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:18:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.412252427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.361994316s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:18:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.362023051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.401889435s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:17:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.401917839s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.421487496s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:16:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.421547962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.379294348s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:15:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.379320841s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.365516776s trace: https://jaeger.dev.storj.io/trace/6b308ae490519c20 FAIL (2025-11-04 09:15:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.365594184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.931977568s trace: https://jaeger.dev.storj.io/trace/1d7439731666bbaf FAIL (2025-11-04 09:14:43 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.932019468s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)