Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X.....X..XX] duration: 30.004623867s trace: https://jaeger.dev.storj.io/trace/663b882f054b9bba FAIL (2025-12-30 04:51:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: PTH2D3KRDECYNR5YPK7L2CMG662EHDHEYTDISRV42GUCOAQ2K5NQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: N4VAQ47QMBCIKDKSJSJEZO322ARGIZPZHGX4IOJ6FIYH4N2Y3V4Q) context deadline exceeded ======LOGS====== Dec 30 04:51:21.934069048 (589h22m21.805152093s) [upload=516657]: Starting upload Dec 30 04:51:21.934167225 (589h22m21.80525027s) [upload=516657]: Done starting upload Dec 30 04:51:21.934474310 (589h22m21.805557355s) [upload=516657]: Uploading segments... Dec 30 04:51:21.934524856 (589h22m21.805607891s) [upload=516657]: Got next segment. Inline: false Dec 30 04:51:21.934559682 (589h22m21.805642737s) [upload=516657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:51:21.934566025 (589h22m21.80564907s) [upload=516657]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:51:22.183159878 (589h22m22.054242913s) [seg_pos={0 0}, upload=516657]: Begin upload segment... Dec 30 04:51:22.183212789 (589h22m22.054295824s) [seg_pos={0 0}, upload=516657]: Done begin upload segment. Dec 30 04:51:22.183237155 (589h22m22.05432019s) [noise=true , address=145.79.227.10:4402 , piece_id=E3HORGTDJPMPJGD3VEMEDGSXLH3BW3N4YNKTY5DCKLCE3FZWGKYQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.183394184 (589h22m22.054477229s) [noise=true , address=145.79.227.16:4912 , piece_id=WIJN3HTMI3PRTLMNKNDH7AEGUONECRD6CZSCYUKQLJKCXXU7H3VA, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.184167747 (589h22m22.055250782s) [noise=true , address=145.79.227.22:4509 , piece_id=TAIZLA2ENMQCFDLBLA6TQGSZ2TSE4JTWND7ZJSAH7FN3RTRSCZ7A, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.184327241 (589h22m22.055410276s) [noise=true , address=145.79.227.11:4732 , piece_id=VD5YZHCXZXU4SLPCFJCVQ74YQFI3MN634HVN3TQV4I7HMDRYQ6FQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.185204060 (589h22m22.056287105s) [noise=true , address=77.90.31.14:4661 , piece_id=OJNEAFV7I3OQ332KZRG75PSL6YRGMI2CXIBNVZFMYL5HI3WQ6FHQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.185226873 (589h22m22.056309918s) [noise=true , address=145.79.227.19:4077 , piece_id=N6ZHDJPMDI7N5CIHCP2WDTHZUPLETD3M7KXVCKVBI3RTIIURTJBQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.185839220 (589h22m22.056922255s) [noise=true , address=45.91.116.8:4046 , piece_id=PTH2D3KRDECYNR5YPK7L2CMG662EHDHEYTDISRV42GUCOAQ2K5NQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.186771956 (589h22m22.057855001s) [noise=true , address=145.79.227.35:4482 , piece_id=QEVYKA4HWZGZJAQZ27BX2LOSYWZVQB3YLRY4TLUO6SQWMHPCLIGA, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.186952039 (589h22m22.058035084s) [noise=true , address=77.90.31.12:4589 , piece_id=TKEKZEVJ4SUEMWB7FUDEH6PIOLQI6ENQ5BTQIWU5TMHE33RXX5AQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.188121516 (589h22m22.059204551s) [noise=true , address=145.79.227.32:4836 , piece_id=6445UIFIOXOUBPTJ5HDCVBKFJYUQM7INKETH3ABSWFXPL5R7TAVA, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.189033362 (589h22m22.060116407s) [noise=true , address=77.90.31.21:4878 , piece_id=H77UVTHT5QXYO5NIR4DGLUDVBQXXKFSBUYRBMWOZ22CXEWXMTSFA, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.189453262 (589h22m22.060536297s) [noise=true , address=45.91.116.4:4939 , piece_id=QYINM3QAQWZZGNAYO5JB7DKX2E2P57FLRK7US4Z7DSCO2Q24EVBQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.190234509 (589h22m22.061317554s) [noise=true , address=145.79.227.28:4727 , piece_id=2SVAJEEJMFFVUN54AKIUSDM5JZR6YJ5E44UYEL6RY3G5YE6EN5ZQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.190927609 (589h22m22.062010654s) [noise=true , address=145.79.227.23:4547 , piece_id=J45EIJ37AYYIJOQZD3EOIVQGO5YFAWA233M5D7DJFBHCQYF2LOIA, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.191830778 (589h22m22.062913823s) [noise=true , address=45.91.116.7:4009 , piece_id=N4VAQ47QMBCIKDKSJSJEZO322ARGIZPZHGX4IOJ6FIYH4N2Y3V4Q, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:22.192800345 (589h22m22.06388338s) [noise=true , address=145.79.227.52:4447 , piece_id=OFTB3AIQVMJ7SLSE2UJMEJM66EQWPWPJKHWRXJ3HMZFK54UMXVQQ, seg_pos={0 0}, upload=516657]: Uploading piece... Dec 30 04:51:23.890651089 (589h22m23.761734134s) [upload=516657]: Next returned nil segment Dec 30 04:51:23.890666277 (589h22m23.761749323s) [upload=516657]: Waiting for error group managing segments... Dec 30 04:51:24.178247149 (589h22m24.049330184s) [noise=true , address=145.79.227.52:4447 , piece_id=OFTB3AIQVMJ7SLSE2UJMEJM66EQWPWPJKHWRXJ3HMZFK54UMXVQQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.178766026 (589h22m24.049849071s) [noise=true , address=145.79.227.35:4482 , piece_id=QEVYKA4HWZGZJAQZ27BX2LOSYWZVQB3YLRY4TLUO6SQWMHPCLIGA, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.180967619 (589h22m24.052050654s) [noise=true , address=145.79.227.32:4836 , piece_id=6445UIFIOXOUBPTJ5HDCVBKFJYUQM7INKETH3ABSWFXPL5R7TAVA, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.187846387 (589h22m24.058929432s) [noise=true , address=145.79.227.23:4547 , piece_id=J45EIJ37AYYIJOQZD3EOIVQGO5YFAWA233M5D7DJFBHCQYF2LOIA, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.188108486 (589h22m24.059191541s) [noise=true , address=145.79.227.22:4509 , piece_id=TAIZLA2ENMQCFDLBLA6TQGSZ2TSE4JTWND7ZJSAH7FN3RTRSCZ7A, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.188953956 (589h22m24.060036991s) [noise=true , address=145.79.227.28:4727 , piece_id=2SVAJEEJMFFVUN54AKIUSDM5JZR6YJ5E44UYEL6RY3G5YE6EN5ZQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.196922800 (589h22m24.068005835s) [noise=true , address=145.79.227.10:4402 , piece_id=E3HORGTDJPMPJGD3VEMEDGSXLH3BW3N4YNKTY5DCKLCE3FZWGKYQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.197786976 (589h22m24.068870021s) [noise=true , address=145.79.227.16:4912 , piece_id=WIJN3HTMI3PRTLMNKNDH7AEGUONECRD6CZSCYUKQLJKCXXU7H3VA, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.205888302 (589h22m24.076971337s) [noise=true , address=145.79.227.11:4732 , piece_id=VD5YZHCXZXU4SLPCFJCVQ74YQFI3MN634HVN3TQV4I7HMDRYQ6FQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.214213674 (589h22m24.085296709s) [noise=true , address=145.79.227.19:4077 , piece_id=N6ZHDJPMDI7N5CIHCP2WDTHZUPLETD3M7KXVCKVBI3RTIIURTJBQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.443384153 (589h22m24.314467188s) [noise=true , address=45.91.116.8:4046 , piece_id=PTH2D3KRDECYNR5YPK7L2CMG662EHDHEYTDISRV42GUCOAQ2K5NQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.448425182 (589h22m24.319508227s) [noise=true , address=45.91.116.7:4009 , piece_id=N4VAQ47QMBCIKDKSJSJEZO322ARGIZPZHGX4IOJ6FIYH4N2Y3V4Q, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.475058335 (589h22m24.34614137s) [noise=true , address=45.91.116.4:4939 , piece_id=QYINM3QAQWZZGNAYO5JB7DKX2E2P57FLRK7US4Z7DSCO2Q24EVBQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: <nil> Dec 30 04:51:24.475078764 (589h22m24.3461618s) [seg_pos={0 0}, upload=516657]: Segment reached optimal threshold of 13 pieces. Dec 30 04:51:24.475265670 (589h22m24.346348705s) [noise=true , address=77.90.31.14:4661 , piece_id=OJNEAFV7I3OQ332KZRG75PSL6YRGMI2CXIBNVZFMYL5HI3WQ6FHQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:24.475273695 (589h22m24.34635673s) [noise=true , address=77.90.31.14:4661 , piece_id=OJNEAFV7I3OQ332KZRG75PSL6YRGMI2CXIBNVZFMYL5HI3WQ6FHQ, seg_pos={0 0}, upload=516657]: Long tail cancellation due to optimal threshold Dec 30 04:51:24.475387122 (589h22m24.346470167s) [noise=true , address=77.90.31.21:4878 , piece_id=H77UVTHT5QXYO5NIR4DGLUDVBQXXKFSBUYRBMWOZ22CXEWXMTSFA, seg_pos={0 0}, upload=516657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:24.475394115 (589h22m24.34647716s) [noise=true , address=77.90.31.21:4878 , piece_id=H77UVTHT5QXYO5NIR4DGLUDVBQXXKFSBUYRBMWOZ22CXEWXMTSFA, seg_pos={0 0}, upload=516657]: Long tail cancellation due to optimal threshold Dec 30 04:51:24.475606098 (589h22m24.346689154s) [noise=true , address=77.90.31.12:4589 , piece_id=TKEKZEVJ4SUEMWB7FUDEH6PIOLQI6ENQ5BTQIWU5TMHE33RXX5AQ, seg_pos={0 0}, upload=516657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:24.475617079 (589h22m24.346700144s) [noise=true , address=77.90.31.12:4589 , piece_id=TKEKZEVJ4SUEMWB7FUDEH6PIOLQI6ENQ5BTQIWU5TMHE33RXX5AQ, seg_pos={0 0}, upload=516657]: Long tail cancellation due to optimal threshold Dec 30 04:51:24.475639913 (589h22m24.346722958s) [seg_pos={0 0}, upload=516657]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:51:24.475719204 (589h22m24.346802249s) [upload=516657]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:24.475725455 (589h22m24.3468085s) [upload=516657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:24.791592909 (589h22m24.662675954s) [upload=516657]: Done uploading segments... Dec 30 04:51:24.791616784 (589h22m24.662699829s) [upload=516657]: Upload finished. err: <nil> Duration: 30.004647482s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.....X..............X................X.............X.......] duration: 3.445591109s trace: https://jaeger.dev.storj.io/trace/77f156dfc046b501 FAIL (2025-12-30 04:51:35 UTC) orbiter warning: context deadline exceeded Duration: 3.445616287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.....X.....] duration: 48.974697742s trace: https://jaeger.dev.storj.io/trace/5a71d1d7e99c0ec6 FAIL (2025-12-30 04:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.576312321 (589h17m42.447395366s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.577549918 (589h17m42.448632953s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.578974851 (589h17m42.450057886s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.580367783 (589h17m42.451450818s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.581567388 (589h17m42.452650423s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.582814021 (589h17m42.453897056s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.584002213 (589h17m42.455085258s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.585288813 (589h17m42.456371858s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.586586455 (589h17m42.4576695s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.588133640 (589h17m42.459216675s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.589628978 (589h17m42.460712013s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591216792 (589h17m42.462299827s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591966859 (589h17m42.463049894s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593499888 (589h17m42.464582924s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593996395 (589h17m42.46507944s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:44.048371258 (589h17m43.919454293s) [upload=516588]: Next returned nil segment Dec 30 04:46:44.048384013 (589h17m43.919467058s) [upload=516588]: Waiting for error group managing segments... Dec 30 04:46:44.447700633 (589h17m44.318783678s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.453673536 (589h17m44.324756572s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.458257975 (589h17m44.32934101s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.463944624 (589h17m44.335027659s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.469691828 (589h17m44.340774863s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.688602448 (589h17m44.559685493s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.048959672 (589h17m44.920042718s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.199176519 (589h17m45.070259554s) [noise=true , address=77.90.31.21:4893 , piece_id=C4E5JVOZ4Z5MBCYVCBCWBNMYWDMFPURSWCGXUXUIMKY3R7ZU4ELA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364723747 (589h17m45.235806782s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364814078 (589h17m45.235897123s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.383057617 (589h17m45.254140662s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.408958516 (589h17m45.280041551s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.413279004 (589h17m45.284362049s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467102871 (589h17m45.338185906s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467128300 (589h17m45.338211335s) [seg_pos={0 0}, upload=516588]: Segment reached optimal threshold of 14 pieces. Dec 30 04:46:45.467342818 (589h17m45.338425863s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467352838 (589h17m45.338435883s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467724515 (589h17m45.33880756s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467733774 (589h17m45.338816829s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467897455 (589h17m45.33898049s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection Dec 30 04:46:45.467905209 (589h17m45.338988254s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468253593 (589h17m45.339336638s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: read tcp 188.166.225.170:56378->77.90.31.11:4588: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:46:45.468263031 (589h17m45.339346076s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468282408 (589h17m45.339365453s) [seg_pos={0 0}, upload=516588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:46:45.468368562 (589h17m45.339451607s) [upload=516588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:46:45.468374774 (589h17m45.339457819s) [upload=516588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:46:45.755910376 (589h17m45.626993421s) [upload=516588]: Done uploading segments... Dec 30 04:46:45.755950452 (589h17m45.627033497s) [upload=516588]: Upload finished. err: <nil> Duration: 48.97471248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X.....X..X] duration: 30.002902905s trace: https://jaeger.dev.storj.io/trace/3380da2ffcfe26a9 FAIL (2025-12-30 04:50:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 30 04:50:14.450019534 (589h21m14.321102569s) [upload=516641]: Got next segment. Inline: false Dec 30 04:50:14.450064490 (589h21m14.321147535s) [upload=516641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:50:14.450069139 (589h21m14.321152184s) [upload=516641]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:50:14.717268238 (589h21m14.588351283s) [seg_pos={0 0}, upload=516641]: Begin upload segment... Dec 30 04:50:14.717320428 (589h21m14.588403473s) [seg_pos={0 0}, upload=516641]: Done begin upload segment. Dec 30 04:50:14.717354893 (589h21m14.588437928s) [noise=true , address=145.79.227.46:4603 , piece_id=SKU2TAR4L52TT2T4RDN7ECM5EKXQQUPVWEIIIVWYO3RDGQGK7DIQ, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.717370613 (589h21m14.588453648s) [noise=true , address=145.79.227.3:4149 , piece_id=F7VK25SPJI5SYUAH4OIFTFW7TSST22DNXL4HGWS6SQNATNGTCJRQ, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.717746949 (589h21m14.588829985s) [noise=true , address=45.91.116.12:4227 , piece_id=7UJ2PMZMYPJF6JTWQWJKIMMHAEKDV5NS7QCVAYH5BLCO5DFAF2EA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.718044105 (589h21m14.58912714s) [noise=true , address=145.79.227.51:4409 , piece_id=JKH2JBUJ72ILI4YQU4JZLRXUMLB62LFA6SSWENKGLAIYR3ZOCCAA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.718389392 (589h21m14.589472427s) [noise=true , address=145.79.227.5:4222 , piece_id=GOVPML7F7AZUWG2HYA33F3STMHXQCU7JRHGX6ZO5BP5FYCEYPRSA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.718676338 (589h21m14.589759374s) [noise=true , address=145.79.227.17:4971 , piece_id=IPLTRBRRADG73YNTZNVSTALSXBMMQ3VG2GQ2M2F6MCZ53AHAE3AA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.719041424 (589h21m14.590124459s) [noise=true , address=145.79.227.32:4836 , piece_id=63TBVROWNASUDF2OBFJOPDBDFRPF34QGWLNVFBOIWDYAQZSBKQMA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.719400758 (589h21m14.590483803s) [noise=true , address=145.79.227.27:4689 , piece_id=KQ326YNZRCKGBTKJOBO2KITV3GWJSNPTPUTO4GAGYDNIOL7X7G2A, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.719727331 (589h21m14.590810376s) [noise=true , address=145.79.227.41:4087 , piece_id=JNRGJRZ5TYFF7F2WUNOOD36DFF2KHLWW54RAIGXHNXC5BWKBW6LA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.720071536 (589h21m14.591154571s) [noise=true , address=145.79.227.30:4762 , piece_id=PSH6M6GGKEJ7PJILIFJ5MJQGGRXT5R4VHBNZTUMB577UQ2RSFEQA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.721127557 (589h21m14.592210592s) [noise=true , address=145.79.227.36:4517 , piece_id=DZXZ6DQ3IVPQTQWFMLTUGEFPYIPJ45HGYGHQLIZPJ75WPNO3S6JQ, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.722256275 (589h21m14.593339321s) [noise=true , address=145.79.227.48:4301 , piece_id=VRVJB7IVD4WFQJEUTVEEAGLBGXTJS22UFYBSQDQ2OOZCWVROVZSA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.723700195 (589h21m14.594783231s) [noise=true , address=145.79.227.24:4582 , piece_id=UG7CZU27REVAXU47JYJQQRXRHT2FL2AWGG2YMFBISCRHGOMVVABQ, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.726522089 (589h21m14.597605124s) [noise=true , address=77.90.31.14:4663 , piece_id=UT3K2QHURVV25IWUQWIRXNMT4AT7KQJIF63QI3DDH7BBKKAHUI4Q, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.730560809 (589h21m14.601643844s) [noise=true , address=77.90.31.19:4844 , piece_id=VURICOQIIYSWLVGA5QQBUXZIGU7F2GSZNJAY5HVGMRAMPDGDWTTA, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:14.732671599 (589h21m14.603754644s) [noise=true , address=45.91.116.11:4119 , piece_id=STPTGUWCNNOX4FYTLWPCDG6S5QRESZBRCLTQVJEUVRVNK4MPLHEQ, seg_pos={0 0}, upload=516641]: Uploading piece... Dec 30 04:50:16.336999703 (589h21m16.208082738s) [upload=516641]: Next returned nil segment Dec 30 04:50:16.337018459 (589h21m16.208101504s) [upload=516641]: Waiting for error group managing segments... Dec 30 04:50:16.716616716 (589h21m16.587699751s) [noise=true , address=145.79.227.51:4409 , piece_id=JKH2JBUJ72ILI4YQU4JZLRXUMLB62LFA6SSWENKGLAIYR3ZOCCAA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.719055209 (589h21m16.590138254s) [noise=true , address=145.79.227.46:4603 , piece_id=SKU2TAR4L52TT2T4RDN7ECM5EKXQQUPVWEIIIVWYO3RDGQGK7DIQ, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.721637045 (589h21m16.59272008s) [noise=true , address=145.79.227.5:4222 , piece_id=GOVPML7F7AZUWG2HYA33F3STMHXQCU7JRHGX6ZO5BP5FYCEYPRSA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.722583357 (589h21m16.593666402s) [noise=true , address=145.79.227.17:4971 , piece_id=IPLTRBRRADG73YNTZNVSTALSXBMMQ3VG2GQ2M2F6MCZ53AHAE3AA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.723145136 (589h21m16.594228181s) [noise=true , address=145.79.227.24:4582 , piece_id=UG7CZU27REVAXU47JYJQQRXRHT2FL2AWGG2YMFBISCRHGOMVVABQ, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.723937896 (589h21m16.595020941s) [noise=true , address=145.79.227.36:4517 , piece_id=DZXZ6DQ3IVPQTQWFMLTUGEFPYIPJ45HGYGHQLIZPJ75WPNO3S6JQ, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.724446305 (589h21m16.59552935s) [noise=true , address=145.79.227.32:4836 , piece_id=63TBVROWNASUDF2OBFJOPDBDFRPF34QGWLNVFBOIWDYAQZSBKQMA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.733603371 (589h21m16.604686416s) [noise=true , address=145.79.227.41:4087 , piece_id=JNRGJRZ5TYFF7F2WUNOOD36DFF2KHLWW54RAIGXHNXC5BWKBW6LA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.738853117 (589h21m16.609936162s) [noise=true , address=145.79.227.48:4301 , piece_id=VRVJB7IVD4WFQJEUTVEEAGLBGXTJS22UFYBSQDQ2OOZCWVROVZSA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.745127174 (589h21m16.616210209s) [noise=true , address=145.79.227.27:4689 , piece_id=KQ326YNZRCKGBTKJOBO2KITV3GWJSNPTPUTO4GAGYDNIOL7X7G2A, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.757456280 (589h21m16.628539325s) [noise=true , address=145.79.227.3:4149 , piece_id=F7VK25SPJI5SYUAH4OIFTFW7TSST22DNXL4HGWS6SQNATNGTCJRQ, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:16.930949720 (589h21m16.802032755s) [noise=true , address=45.91.116.12:4227 , piece_id=7UJ2PMZMYPJF6JTWQWJKIMMHAEKDV5NS7QCVAYH5BLCO5DFAF2EA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:17.014749408 (589h21m16.885832443s) [noise=true , address=77.90.31.19:4844 , piece_id=VURICOQIIYSWLVGA5QQBUXZIGU7F2GSZNJAY5HVGMRAMPDGDWTTA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: <nil> Dec 30 04:50:17.014773834 (589h21m16.885856879s) [seg_pos={0 0}, upload=516641]: Segment reached optimal threshold of 13 pieces. Dec 30 04:50:17.014978043 (589h21m16.886061078s) [noise=true , address=145.79.227.30:4762 , piece_id=PSH6M6GGKEJ7PJILIFJ5MJQGGRXT5R4VHBNZTUMB577UQ2RSFEQA, seg_pos={0 0}, upload=516641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 04:50:17.014984605 (589h21m16.88606765s) [noise=true , address=145.79.227.30:4762 , piece_id=PSH6M6GGKEJ7PJILIFJ5MJQGGRXT5R4VHBNZTUMB577UQ2RSFEQA, seg_pos={0 0}, upload=516641]: Long tail cancellation due to optimal threshold Dec 30 04:50:17.015126245 (589h21m16.88620929s) [noise=true , address=45.91.116.11:4119 , piece_id=STPTGUWCNNOX4FYTLWPCDG6S5QRESZBRCLTQVJEUVRVNK4MPLHEQ, seg_pos={0 0}, upload=516641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:43456->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:43456->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43456->45.91.116.11:4119: use of closed network connection Dec 30 04:50:17.015135113 (589h21m16.886218158s) [noise=true , address=45.91.116.11:4119 , piece_id=STPTGUWCNNOX4FYTLWPCDG6S5QRESZBRCLTQVJEUVRVNK4MPLHEQ, seg_pos={0 0}, upload=516641]: Long tail cancellation due to optimal threshold Dec 30 04:50:17.015196228 (589h21m16.886279293s) [noise=true , address=77.90.31.14:4663 , piece_id=UT3K2QHURVV25IWUQWIRXNMT4AT7KQJIF63QI3DDH7BBKKAHUI4Q, seg_pos={0 0}, upload=516641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:59168->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:59168->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->77.90.31.14:4663: use of closed network connection Dec 30 04:50:17.015208312 (589h21m16.886291357s) [noise=true , address=77.90.31.14:4663 , piece_id=UT3K2QHURVV25IWUQWIRXNMT4AT7KQJIF63QI3DDH7BBKKAHUI4Q, seg_pos={0 0}, upload=516641]: Long tail cancellation due to optimal threshold Dec 30 04:50:17.015261262 (589h21m16.886344307s) [seg_pos={0 0}, upload=516641]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:50:17.015360311 (589h21m16.886443346s) [upload=516641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:50:17.015366833 (589h21m16.886449868s) [upload=516641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:50:17.340651077 (589h21m17.211734122s) [upload=516641]: Done uploading segments... Dec 30 04:50:17.340674382 (589h21m17.211757417s) [upload=516641]: Upload finished. err: <nil> Duration: 30.002913635s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.....X..............X................X.............X......] duration: 3.787460212s trace: https://jaeger.dev.storj.io/trace/77f156dfc046b501 FAIL (2025-12-30 04:50:40 UTC) orbiter warning: context deadline exceeded Duration: 3.787484528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.....X....] duration: 41.890779211s trace: https://jaeger.dev.storj.io/trace/5a71d1d7e99c0ec6 FAIL (2025-12-30 04:50:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.576312321 (589h17m42.447395366s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.577549918 (589h17m42.448632953s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.578974851 (589h17m42.450057886s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.580367783 (589h17m42.451450818s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.581567388 (589h17m42.452650423s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.582814021 (589h17m42.453897056s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.584002213 (589h17m42.455085258s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.585288813 (589h17m42.456371858s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.586586455 (589h17m42.4576695s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.588133640 (589h17m42.459216675s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.589628978 (589h17m42.460712013s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591216792 (589h17m42.462299827s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591966859 (589h17m42.463049894s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593499888 (589h17m42.464582924s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593996395 (589h17m42.46507944s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:44.048371258 (589h17m43.919454293s) [upload=516588]: Next returned nil segment Dec 30 04:46:44.048384013 (589h17m43.919467058s) [upload=516588]: Waiting for error group managing segments... Dec 30 04:46:44.447700633 (589h17m44.318783678s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.453673536 (589h17m44.324756572s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.458257975 (589h17m44.32934101s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.463944624 (589h17m44.335027659s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.469691828 (589h17m44.340774863s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.688602448 (589h17m44.559685493s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.048959672 (589h17m44.920042718s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.199176519 (589h17m45.070259554s) [noise=true , address=77.90.31.21:4893 , piece_id=C4E5JVOZ4Z5MBCYVCBCWBNMYWDMFPURSWCGXUXUIMKY3R7ZU4ELA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364723747 (589h17m45.235806782s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364814078 (589h17m45.235897123s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.383057617 (589h17m45.254140662s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.408958516 (589h17m45.280041551s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.413279004 (589h17m45.284362049s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467102871 (589h17m45.338185906s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467128300 (589h17m45.338211335s) [seg_pos={0 0}, upload=516588]: Segment reached optimal threshold of 14 pieces. Dec 30 04:46:45.467342818 (589h17m45.338425863s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467352838 (589h17m45.338435883s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467724515 (589h17m45.33880756s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467733774 (589h17m45.338816829s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467897455 (589h17m45.33898049s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection Dec 30 04:46:45.467905209 (589h17m45.338988254s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468253593 (589h17m45.339336638s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: read tcp 188.166.225.170:56378->77.90.31.11:4588: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:46:45.468263031 (589h17m45.339346076s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468282408 (589h17m45.339365453s) [seg_pos={0 0}, upload=516588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:46:45.468368562 (589h17m45.339451607s) [upload=516588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:46:45.468374774 (589h17m45.339457819s) [upload=516588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:46:45.755910376 (589h17m45.626993421s) [upload=516588]: Done uploading segments... Dec 30 04:46:45.755950452 (589h17m45.627033497s) [upload=516588]: Upload finished. err: <nil> Duration: 41.890797526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X.....X..............X................X.............X.....] duration: 3.532989657s trace: https://jaeger.dev.storj.io/trace/77f156dfc046b501 FAIL (2025-12-30 04:50:14 UTC) orbiter warning: context deadline exceeded Duration: 3.533020095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X.....X..] duration: 22.756177902s trace: https://jaeger.dev.storj.io/trace/30b70441b5285795 FAIL (2025-12-30 04:50:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:48:16.351254721 (589h19m16.222337756s) [upload=516612]: Starting upload Dec 30 04:48:16.351310277 (589h19m16.222393312s) [upload=516612]: Done starting upload Dec 30 04:48:16.351505728 (589h19m16.222588773s) [upload=516612]: Uploading segments... Dec 30 04:48:16.351918705 (589h19m16.22300174s) [upload=516612]: Got next segment. Inline: false Dec 30 04:48:16.352241449 (589h19m16.223324494s) [upload=516612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:48:16.352251659 (589h19m16.223334694s) [upload=516612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:48:16.598503372 (589h19m16.469586417s) [seg_pos={0 0}, upload=516612]: Begin upload segment... Dec 30 04:48:16.598562143 (589h19m16.469645178s) [seg_pos={0 0}, upload=516612]: Done begin upload segment. Dec 30 04:48:16.598595998 (589h19m16.469679043s) [noise=true , address=145.79.227.48:4303 , piece_id=YZZDMASYTFQHD7ZG73TF3ADRU43GJJKMUEA2EJ57Y4JBNYXBY5IA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.598602190 (589h19m16.469685235s) [noise=true , address=145.79.227.50:4373 , piece_id=ZOSEM57BPC2K46HZPPDYZ6GS4T6CZSVPV73FATPTDFZLHTR7WKFA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.598771762 (589h19m16.469854807s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.599334805 (589h19m16.47041786s) [noise=true , address=145.79.227.3:4149 , piece_id=6BK4NNTFBARK5B6GP5BKJJ2EGFZVKMAHBNISHTRJ4KLXGGAL7NEQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.599950757 (589h19m16.471033803s) [noise=true , address=45.91.116.9:4263 , piece_id=MS63WCIL6KD52UR7KB6PWIWEMRMEOKFTGCPRXDP7ALH2V3B65N3Q, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.600359506 (589h19m16.471442551s) [noise=true , address=145.79.227.39:4015 , piece_id=VNBKBYRABMFGH2D45FRPO55GHLEAMR4ZGX6BRNR2WXJXVUO7JMZQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.600756453 (589h19m16.471839488s) [noise=true , address=145.79.227.17:4972 , piece_id=B6E5ZDHFMQVFLNTP7WO5H2K7WYWHIL7VWQ2GECFBYBX7JK77GOPA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.601263748 (589h19m16.472346793s) [noise=true , address=145.79.227.47:4265 , piece_id=GY32IYEEOI6MA77CKR4NGZ5BQ6DZTYUKENKVTJKVFNZSLX5TBUEA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.601746918 (589h19m16.472829953s) [noise=true , address=77.90.31.15:4697 , piece_id=B2MVW3GTX72W2QKUHQJF7UEHF6632WPG6AP27QXTWHYHF66Z3ZLA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.602360086 (589h19m16.473443131s) [noise=true , address=45.91.116.7:4009 , piece_id=3IW5RK5QWLXBWWOHGFXIRIWQICFTMXJIJU7J2CELBMNULLHQ4SLQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.602986319 (589h19m16.474069364s) [noise=true , address=145.79.227.41:4085 , piece_id=XMFMYSG62PECO7Z4KOL54Y3YBI4T6DRYSF3POTBPJK3MH2XPDJHA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.604567419 (589h19m16.475650464s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.604876638 (589h19m16.475959683s) [noise=true , address=77.90.31.16:4735 , piece_id=GLSVHWZHXUAQFQW5R5VXBZAX6PAFSTH2HJAITX2HJGTB2CDW4ZYA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.606250073 (589h19m16.477333118s) [noise=true , address=77.90.31.14:4664 , piece_id=7VZI6AGSFPCIEX75V5LQTMWC3HGN2QJTW3SJJNKG4OBAPOS36X5A, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.606410859 (589h19m16.477493904s) [noise=true , address=145.79.227.15:4873 , piece_id=DEN6TNZFBFM7NQ6X4V24OB5ZYVEC5XTBT4FG2RD6F5DVQ5XMQETQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.607948016 (589h19m16.479031061s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:18.311216546 (589h19m18.182299611s) [upload=516612]: Next returned nil segment Dec 30 04:48:18.311234000 (589h19m18.182317055s) [upload=516612]: Waiting for error group managing segments... Dec 30 04:48:18.591742483 (589h19m18.462825528s) [noise=true , address=145.79.227.48:4303 , piece_id=YZZDMASYTFQHD7ZG73TF3ADRU43GJJKMUEA2EJ57Y4JBNYXBY5IA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.602695660 (589h19m18.473778695s) [noise=true , address=145.79.227.15:4873 , piece_id=DEN6TNZFBFM7NQ6X4V24OB5ZYVEC5XTBT4FG2RD6F5DVQ5XMQETQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.608425811 (589h19m18.479508846s) [noise=true , address=145.79.227.41:4085 , piece_id=XMFMYSG62PECO7Z4KOL54Y3YBI4T6DRYSF3POTBPJK3MH2XPDJHA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.608505223 (589h19m18.479588268s) [noise=true , address=145.79.227.50:4373 , piece_id=ZOSEM57BPC2K46HZPPDYZ6GS4T6CZSVPV73FATPTDFZLHTR7WKFA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.610582157 (589h19m18.481665202s) [noise=true , address=145.79.227.3:4149 , piece_id=6BK4NNTFBARK5B6GP5BKJJ2EGFZVKMAHBNISHTRJ4KLXGGAL7NEQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.611939562 (589h19m18.483022607s) [noise=true , address=145.79.227.17:4972 , piece_id=B6E5ZDHFMQVFLNTP7WO5H2K7WYWHIL7VWQ2GECFBYBX7JK77GOPA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.613413028 (589h19m18.484496063s) [noise=true , address=145.79.227.47:4265 , piece_id=GY32IYEEOI6MA77CKR4NGZ5BQ6DZTYUKENKVTJKVFNZSLX5TBUEA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.614726989 (589h19m18.485810024s) [noise=true , address=145.79.227.39:4015 , piece_id=VNBKBYRABMFGH2D45FRPO55GHLEAMR4ZGX6BRNR2WXJXVUO7JMZQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.820948820 (589h19m18.692031855s) [noise=true , address=45.91.116.7:4009 , piece_id=3IW5RK5QWLXBWWOHGFXIRIWQICFTMXJIJU7J2CELBMNULLHQ4SLQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.848491565 (589h19m18.7195746s) [noise=true , address=45.91.116.9:4263 , piece_id=MS63WCIL6KD52UR7KB6PWIWEMRMEOKFTGCPRXDP7ALH2V3B65N3Q, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.851265407 (589h19m18.722348442s) [noise=true , address=77.90.31.15:4697 , piece_id=B2MVW3GTX72W2QKUHQJF7UEHF6632WPG6AP27QXTWHYHF66Z3ZLA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.852431679 (589h19m18.723514724s) [noise=true , address=77.90.31.14:4664 , piece_id=7VZI6AGSFPCIEX75V5LQTMWC3HGN2QJTW3SJJNKG4OBAPOS36X5A, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.894615313 (589h19m18.765698358s) [noise=true , address=77.90.31.16:4735 , piece_id=GLSVHWZHXUAQFQW5R5VXBZAX6PAFSTH2HJAITX2HJGTB2CDW4ZYA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.894661951 (589h19m18.765745016s) [seg_pos={0 0}, upload=516612]: Segment reached optimal threshold of 13 pieces. Dec 30 04:48:18.895031365 (589h19m18.76611441s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895045041 (589h19m18.766128096s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895095036 (589h19m18.766178081s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895111086 (589h19m18.766194132s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895452186 (589h19m18.766535231s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895462135 (589h19m18.76654518s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895487904 (589h19m18.766570949s) [seg_pos={0 0}, upload=516612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:48:18.895619725 (589h19m18.76670276s) [upload=516612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:48:18.895626328 (589h19m18.766709363s) [upload=516612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:48:19.191139763 (589h19m19.062222808s) [upload=516612]: Done uploading segments... Dec 30 04:48:19.191195249 (589h19m19.062278294s) [upload=516612]: Upload finished. err: <nil> Duration: 22.756202618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.....X...] duration: 26.144244939s trace: https://jaeger.dev.storj.io/trace/5a71d1d7e99c0ec6 FAIL (2025-12-30 04:49:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.576312321 (589h17m42.447395366s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.577549918 (589h17m42.448632953s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.578974851 (589h17m42.450057886s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.580367783 (589h17m42.451450818s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.581567388 (589h17m42.452650423s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.582814021 (589h17m42.453897056s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.584002213 (589h17m42.455085258s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.585288813 (589h17m42.456371858s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.586586455 (589h17m42.4576695s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.588133640 (589h17m42.459216675s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.589628978 (589h17m42.460712013s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591216792 (589h17m42.462299827s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591966859 (589h17m42.463049894s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593499888 (589h17m42.464582924s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593996395 (589h17m42.46507944s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:44.048371258 (589h17m43.919454293s) [upload=516588]: Next returned nil segment Dec 30 04:46:44.048384013 (589h17m43.919467058s) [upload=516588]: Waiting for error group managing segments... Dec 30 04:46:44.447700633 (589h17m44.318783678s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.453673536 (589h17m44.324756572s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.458257975 (589h17m44.32934101s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.463944624 (589h17m44.335027659s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.469691828 (589h17m44.340774863s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.688602448 (589h17m44.559685493s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.048959672 (589h17m44.920042718s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.199176519 (589h17m45.070259554s) [noise=true , address=77.90.31.21:4893 , piece_id=C4E5JVOZ4Z5MBCYVCBCWBNMYWDMFPURSWCGXUXUIMKY3R7ZU4ELA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364723747 (589h17m45.235806782s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364814078 (589h17m45.235897123s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.383057617 (589h17m45.254140662s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.408958516 (589h17m45.280041551s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.413279004 (589h17m45.284362049s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467102871 (589h17m45.338185906s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467128300 (589h17m45.338211335s) [seg_pos={0 0}, upload=516588]: Segment reached optimal threshold of 14 pieces. Dec 30 04:46:45.467342818 (589h17m45.338425863s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467352838 (589h17m45.338435883s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467724515 (589h17m45.33880756s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467733774 (589h17m45.338816829s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467897455 (589h17m45.33898049s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection Dec 30 04:46:45.467905209 (589h17m45.338988254s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468253593 (589h17m45.339336638s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: read tcp 188.166.225.170:56378->77.90.31.11:4588: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:46:45.468263031 (589h17m45.339346076s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468282408 (589h17m45.339365453s) [seg_pos={0 0}, upload=516588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:46:45.468368562 (589h17m45.339451607s) [upload=516588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:46:45.468374774 (589h17m45.339457819s) [upload=516588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:46:45.755910376 (589h17m45.626993421s) [upload=516588]: Done uploading segments... Dec 30 04:46:45.755950452 (589h17m45.627033497s) [upload=516588]: Upload finished. err: <nil> Duration: 26.144273193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X..X.....X..............X................X.............X....] duration: 3.452291819s trace: https://jaeger.dev.storj.io/trace/77f156dfc046b501 FAIL (2025-12-30 04:49:36 UTC) orbiter warning: context deadline exceeded Duration: 3.452321835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X.....X.] duration: 5.951830824s trace: https://jaeger.dev.storj.io/trace/30b70441b5285795 FAIL (2025-12-30 04:49:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:48:16.351254721 (589h19m16.222337756s) [upload=516612]: Starting upload Dec 30 04:48:16.351310277 (589h19m16.222393312s) [upload=516612]: Done starting upload Dec 30 04:48:16.351505728 (589h19m16.222588773s) [upload=516612]: Uploading segments... Dec 30 04:48:16.351918705 (589h19m16.22300174s) [upload=516612]: Got next segment. Inline: false Dec 30 04:48:16.352241449 (589h19m16.223324494s) [upload=516612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:48:16.352251659 (589h19m16.223334694s) [upload=516612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:48:16.598503372 (589h19m16.469586417s) [seg_pos={0 0}, upload=516612]: Begin upload segment... Dec 30 04:48:16.598562143 (589h19m16.469645178s) [seg_pos={0 0}, upload=516612]: Done begin upload segment. Dec 30 04:48:16.598595998 (589h19m16.469679043s) [noise=true , address=145.79.227.48:4303 , piece_id=YZZDMASYTFQHD7ZG73TF3ADRU43GJJKMUEA2EJ57Y4JBNYXBY5IA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.598602190 (589h19m16.469685235s) [noise=true , address=145.79.227.50:4373 , piece_id=ZOSEM57BPC2K46HZPPDYZ6GS4T6CZSVPV73FATPTDFZLHTR7WKFA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.598771762 (589h19m16.469854807s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.599334805 (589h19m16.47041786s) [noise=true , address=145.79.227.3:4149 , piece_id=6BK4NNTFBARK5B6GP5BKJJ2EGFZVKMAHBNISHTRJ4KLXGGAL7NEQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.599950757 (589h19m16.471033803s) [noise=true , address=45.91.116.9:4263 , piece_id=MS63WCIL6KD52UR7KB6PWIWEMRMEOKFTGCPRXDP7ALH2V3B65N3Q, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.600359506 (589h19m16.471442551s) [noise=true , address=145.79.227.39:4015 , piece_id=VNBKBYRABMFGH2D45FRPO55GHLEAMR4ZGX6BRNR2WXJXVUO7JMZQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.600756453 (589h19m16.471839488s) [noise=true , address=145.79.227.17:4972 , piece_id=B6E5ZDHFMQVFLNTP7WO5H2K7WYWHIL7VWQ2GECFBYBX7JK77GOPA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.601263748 (589h19m16.472346793s) [noise=true , address=145.79.227.47:4265 , piece_id=GY32IYEEOI6MA77CKR4NGZ5BQ6DZTYUKENKVTJKVFNZSLX5TBUEA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.601746918 (589h19m16.472829953s) [noise=true , address=77.90.31.15:4697 , piece_id=B2MVW3GTX72W2QKUHQJF7UEHF6632WPG6AP27QXTWHYHF66Z3ZLA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.602360086 (589h19m16.473443131s) [noise=true , address=45.91.116.7:4009 , piece_id=3IW5RK5QWLXBWWOHGFXIRIWQICFTMXJIJU7J2CELBMNULLHQ4SLQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.602986319 (589h19m16.474069364s) [noise=true , address=145.79.227.41:4085 , piece_id=XMFMYSG62PECO7Z4KOL54Y3YBI4T6DRYSF3POTBPJK3MH2XPDJHA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.604567419 (589h19m16.475650464s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.604876638 (589h19m16.475959683s) [noise=true , address=77.90.31.16:4735 , piece_id=GLSVHWZHXUAQFQW5R5VXBZAX6PAFSTH2HJAITX2HJGTB2CDW4ZYA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.606250073 (589h19m16.477333118s) [noise=true , address=77.90.31.14:4664 , piece_id=7VZI6AGSFPCIEX75V5LQTMWC3HGN2QJTW3SJJNKG4OBAPOS36X5A, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.606410859 (589h19m16.477493904s) [noise=true , address=145.79.227.15:4873 , piece_id=DEN6TNZFBFM7NQ6X4V24OB5ZYVEC5XTBT4FG2RD6F5DVQ5XMQETQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.607948016 (589h19m16.479031061s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:18.311216546 (589h19m18.182299611s) [upload=516612]: Next returned nil segment Dec 30 04:48:18.311234000 (589h19m18.182317055s) [upload=516612]: Waiting for error group managing segments... Dec 30 04:48:18.591742483 (589h19m18.462825528s) [noise=true , address=145.79.227.48:4303 , piece_id=YZZDMASYTFQHD7ZG73TF3ADRU43GJJKMUEA2EJ57Y4JBNYXBY5IA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.602695660 (589h19m18.473778695s) [noise=true , address=145.79.227.15:4873 , piece_id=DEN6TNZFBFM7NQ6X4V24OB5ZYVEC5XTBT4FG2RD6F5DVQ5XMQETQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.608425811 (589h19m18.479508846s) [noise=true , address=145.79.227.41:4085 , piece_id=XMFMYSG62PECO7Z4KOL54Y3YBI4T6DRYSF3POTBPJK3MH2XPDJHA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.608505223 (589h19m18.479588268s) [noise=true , address=145.79.227.50:4373 , piece_id=ZOSEM57BPC2K46HZPPDYZ6GS4T6CZSVPV73FATPTDFZLHTR7WKFA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.610582157 (589h19m18.481665202s) [noise=true , address=145.79.227.3:4149 , piece_id=6BK4NNTFBARK5B6GP5BKJJ2EGFZVKMAHBNISHTRJ4KLXGGAL7NEQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.611939562 (589h19m18.483022607s) [noise=true , address=145.79.227.17:4972 , piece_id=B6E5ZDHFMQVFLNTP7WO5H2K7WYWHIL7VWQ2GECFBYBX7JK77GOPA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.613413028 (589h19m18.484496063s) [noise=true , address=145.79.227.47:4265 , piece_id=GY32IYEEOI6MA77CKR4NGZ5BQ6DZTYUKENKVTJKVFNZSLX5TBUEA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.614726989 (589h19m18.485810024s) [noise=true , address=145.79.227.39:4015 , piece_id=VNBKBYRABMFGH2D45FRPO55GHLEAMR4ZGX6BRNR2WXJXVUO7JMZQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.820948820 (589h19m18.692031855s) [noise=true , address=45.91.116.7:4009 , piece_id=3IW5RK5QWLXBWWOHGFXIRIWQICFTMXJIJU7J2CELBMNULLHQ4SLQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.848491565 (589h19m18.7195746s) [noise=true , address=45.91.116.9:4263 , piece_id=MS63WCIL6KD52UR7KB6PWIWEMRMEOKFTGCPRXDP7ALH2V3B65N3Q, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.851265407 (589h19m18.722348442s) [noise=true , address=77.90.31.15:4697 , piece_id=B2MVW3GTX72W2QKUHQJF7UEHF6632WPG6AP27QXTWHYHF66Z3ZLA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.852431679 (589h19m18.723514724s) [noise=true , address=77.90.31.14:4664 , piece_id=7VZI6AGSFPCIEX75V5LQTMWC3HGN2QJTW3SJJNKG4OBAPOS36X5A, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.894615313 (589h19m18.765698358s) [noise=true , address=77.90.31.16:4735 , piece_id=GLSVHWZHXUAQFQW5R5VXBZAX6PAFSTH2HJAITX2HJGTB2CDW4ZYA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.894661951 (589h19m18.765745016s) [seg_pos={0 0}, upload=516612]: Segment reached optimal threshold of 13 pieces. Dec 30 04:48:18.895031365 (589h19m18.76611441s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895045041 (589h19m18.766128096s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895095036 (589h19m18.766178081s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895111086 (589h19m18.766194132s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895452186 (589h19m18.766535231s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895462135 (589h19m18.76654518s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895487904 (589h19m18.766570949s) [seg_pos={0 0}, upload=516612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:48:18.895619725 (589h19m18.76670276s) [upload=516612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:48:18.895626328 (589h19m18.766709363s) [upload=516612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:48:19.191139763 (589h19m19.062222808s) [upload=516612]: Done uploading segments... Dec 30 04:48:19.191195249 (589h19m19.062278294s) [upload=516612]: Upload finished. err: <nil> Duration: 5.95185486s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XX..X.....X..............X................X.............X...] duration: 3.380856511s trace: https://jaeger.dev.storj.io/trace/77f156dfc046b501 FAIL (2025-12-30 04:49:00 UTC) orbiter warning: context deadline exceeded Duration: 3.380883803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X.....X] duration: 30.003496019s trace: https://jaeger.dev.storj.io/trace/30b70441b5285795 FAIL (2025-12-30 04:48:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:48:16.351254721 (589h19m16.222337756s) [upload=516612]: Starting upload Dec 30 04:48:16.351310277 (589h19m16.222393312s) [upload=516612]: Done starting upload Dec 30 04:48:16.351505728 (589h19m16.222588773s) [upload=516612]: Uploading segments... Dec 30 04:48:16.351918705 (589h19m16.22300174s) [upload=516612]: Got next segment. Inline: false Dec 30 04:48:16.352241449 (589h19m16.223324494s) [upload=516612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:48:16.352251659 (589h19m16.223334694s) [upload=516612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:48:16.598503372 (589h19m16.469586417s) [seg_pos={0 0}, upload=516612]: Begin upload segment... Dec 30 04:48:16.598562143 (589h19m16.469645178s) [seg_pos={0 0}, upload=516612]: Done begin upload segment. Dec 30 04:48:16.598595998 (589h19m16.469679043s) [noise=true , address=145.79.227.48:4303 , piece_id=YZZDMASYTFQHD7ZG73TF3ADRU43GJJKMUEA2EJ57Y4JBNYXBY5IA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.598602190 (589h19m16.469685235s) [noise=true , address=145.79.227.50:4373 , piece_id=ZOSEM57BPC2K46HZPPDYZ6GS4T6CZSVPV73FATPTDFZLHTR7WKFA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.598771762 (589h19m16.469854807s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.599334805 (589h19m16.47041786s) [noise=true , address=145.79.227.3:4149 , piece_id=6BK4NNTFBARK5B6GP5BKJJ2EGFZVKMAHBNISHTRJ4KLXGGAL7NEQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.599950757 (589h19m16.471033803s) [noise=true , address=45.91.116.9:4263 , piece_id=MS63WCIL6KD52UR7KB6PWIWEMRMEOKFTGCPRXDP7ALH2V3B65N3Q, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.600359506 (589h19m16.471442551s) [noise=true , address=145.79.227.39:4015 , piece_id=VNBKBYRABMFGH2D45FRPO55GHLEAMR4ZGX6BRNR2WXJXVUO7JMZQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.600756453 (589h19m16.471839488s) [noise=true , address=145.79.227.17:4972 , piece_id=B6E5ZDHFMQVFLNTP7WO5H2K7WYWHIL7VWQ2GECFBYBX7JK77GOPA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.601263748 (589h19m16.472346793s) [noise=true , address=145.79.227.47:4265 , piece_id=GY32IYEEOI6MA77CKR4NGZ5BQ6DZTYUKENKVTJKVFNZSLX5TBUEA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.601746918 (589h19m16.472829953s) [noise=true , address=77.90.31.15:4697 , piece_id=B2MVW3GTX72W2QKUHQJF7UEHF6632WPG6AP27QXTWHYHF66Z3ZLA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.602360086 (589h19m16.473443131s) [noise=true , address=45.91.116.7:4009 , piece_id=3IW5RK5QWLXBWWOHGFXIRIWQICFTMXJIJU7J2CELBMNULLHQ4SLQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.602986319 (589h19m16.474069364s) [noise=true , address=145.79.227.41:4085 , piece_id=XMFMYSG62PECO7Z4KOL54Y3YBI4T6DRYSF3POTBPJK3MH2XPDJHA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.604567419 (589h19m16.475650464s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.604876638 (589h19m16.475959683s) [noise=true , address=77.90.31.16:4735 , piece_id=GLSVHWZHXUAQFQW5R5VXBZAX6PAFSTH2HJAITX2HJGTB2CDW4ZYA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.606250073 (589h19m16.477333118s) [noise=true , address=77.90.31.14:4664 , piece_id=7VZI6AGSFPCIEX75V5LQTMWC3HGN2QJTW3SJJNKG4OBAPOS36X5A, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.606410859 (589h19m16.477493904s) [noise=true , address=145.79.227.15:4873 , piece_id=DEN6TNZFBFM7NQ6X4V24OB5ZYVEC5XTBT4FG2RD6F5DVQ5XMQETQ, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:16.607948016 (589h19m16.479031061s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Uploading piece... Dec 30 04:48:18.311216546 (589h19m18.182299611s) [upload=516612]: Next returned nil segment Dec 30 04:48:18.311234000 (589h19m18.182317055s) [upload=516612]: Waiting for error group managing segments... Dec 30 04:48:18.591742483 (589h19m18.462825528s) [noise=true , address=145.79.227.48:4303 , piece_id=YZZDMASYTFQHD7ZG73TF3ADRU43GJJKMUEA2EJ57Y4JBNYXBY5IA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.602695660 (589h19m18.473778695s) [noise=true , address=145.79.227.15:4873 , piece_id=DEN6TNZFBFM7NQ6X4V24OB5ZYVEC5XTBT4FG2RD6F5DVQ5XMQETQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.608425811 (589h19m18.479508846s) [noise=true , address=145.79.227.41:4085 , piece_id=XMFMYSG62PECO7Z4KOL54Y3YBI4T6DRYSF3POTBPJK3MH2XPDJHA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.608505223 (589h19m18.479588268s) [noise=true , address=145.79.227.50:4373 , piece_id=ZOSEM57BPC2K46HZPPDYZ6GS4T6CZSVPV73FATPTDFZLHTR7WKFA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.610582157 (589h19m18.481665202s) [noise=true , address=145.79.227.3:4149 , piece_id=6BK4NNTFBARK5B6GP5BKJJ2EGFZVKMAHBNISHTRJ4KLXGGAL7NEQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.611939562 (589h19m18.483022607s) [noise=true , address=145.79.227.17:4972 , piece_id=B6E5ZDHFMQVFLNTP7WO5H2K7WYWHIL7VWQ2GECFBYBX7JK77GOPA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.613413028 (589h19m18.484496063s) [noise=true , address=145.79.227.47:4265 , piece_id=GY32IYEEOI6MA77CKR4NGZ5BQ6DZTYUKENKVTJKVFNZSLX5TBUEA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.614726989 (589h19m18.485810024s) [noise=true , address=145.79.227.39:4015 , piece_id=VNBKBYRABMFGH2D45FRPO55GHLEAMR4ZGX6BRNR2WXJXVUO7JMZQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.820948820 (589h19m18.692031855s) [noise=true , address=45.91.116.7:4009 , piece_id=3IW5RK5QWLXBWWOHGFXIRIWQICFTMXJIJU7J2CELBMNULLHQ4SLQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.848491565 (589h19m18.7195746s) [noise=true , address=45.91.116.9:4263 , piece_id=MS63WCIL6KD52UR7KB6PWIWEMRMEOKFTGCPRXDP7ALH2V3B65N3Q, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.851265407 (589h19m18.722348442s) [noise=true , address=77.90.31.15:4697 , piece_id=B2MVW3GTX72W2QKUHQJF7UEHF6632WPG6AP27QXTWHYHF66Z3ZLA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.852431679 (589h19m18.723514724s) [noise=true , address=77.90.31.14:4664 , piece_id=7VZI6AGSFPCIEX75V5LQTMWC3HGN2QJTW3SJJNKG4OBAPOS36X5A, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.894615313 (589h19m18.765698358s) [noise=true , address=77.90.31.16:4735 , piece_id=GLSVHWZHXUAQFQW5R5VXBZAX6PAFSTH2HJAITX2HJGTB2CDW4ZYA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: <nil> Dec 30 04:48:18.894661951 (589h19m18.765745016s) [seg_pos={0 0}, upload=516612]: Segment reached optimal threshold of 13 pieces. Dec 30 04:48:18.895031365 (589h19m18.76611441s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895045041 (589h19m18.766128096s) [noise=true , address=45.91.116.5:4300 , piece_id=PXX77B5V2OJZHS3DQ2BHFKPEPLTLQVFXFSHEG5C4AJQG3UAFHEWQ, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895095036 (589h19m18.766178081s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895111086 (589h19m18.766194132s) [noise=true , address=45.91.116.12:4227 , piece_id=IFIE5YCJB57HIQOHWNWG6TL3HHPSH53FFQIN4MXNHTINM24JWGKA, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895452186 (589h19m18.766535231s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 04:48:18.895462135 (589h19m18.76654518s) [noise=true , address=45.91.116.6:4975 , piece_id=RCL4PQZ7Y5SFSJWHBFTPDVX5CYBGWF2RXA32QZ5MKCNT4Y6GNXUQ, seg_pos={0 0}, upload=516612]: Long tail cancellation due to optimal threshold Dec 30 04:48:18.895487904 (589h19m18.766570949s) [seg_pos={0 0}, upload=516612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:48:18.895619725 (589h19m18.76670276s) [upload=516612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:48:18.895626328 (589h19m18.766709363s) [upload=516612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:48:19.191139763 (589h19m19.062222808s) [upload=516612]: Done uploading segments... Dec 30 04:48:19.191195249 (589h19m19.062278294s) [upload=516612]: Upload finished. err: <nil> Duration: 30.00352265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..................................................X.....X..] duration: 15.392071219s trace: https://jaeger.dev.storj.io/trace/5a71d1d7e99c0ec6 FAIL (2025-12-30 04:48:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.576312321 (589h17m42.447395366s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.577549918 (589h17m42.448632953s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.578974851 (589h17m42.450057886s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.580367783 (589h17m42.451450818s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.581567388 (589h17m42.452650423s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.582814021 (589h17m42.453897056s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.584002213 (589h17m42.455085258s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.585288813 (589h17m42.456371858s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.586586455 (589h17m42.4576695s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.588133640 (589h17m42.459216675s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.589628978 (589h17m42.460712013s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591216792 (589h17m42.462299827s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591966859 (589h17m42.463049894s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593499888 (589h17m42.464582924s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593996395 (589h17m42.46507944s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:44.048371258 (589h17m43.919454293s) [upload=516588]: Next returned nil segment Dec 30 04:46:44.048384013 (589h17m43.919467058s) [upload=516588]: Waiting for error group managing segments... Dec 30 04:46:44.447700633 (589h17m44.318783678s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.453673536 (589h17m44.324756572s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.458257975 (589h17m44.32934101s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.463944624 (589h17m44.335027659s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.469691828 (589h17m44.340774863s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.688602448 (589h17m44.559685493s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.048959672 (589h17m44.920042718s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.199176519 (589h17m45.070259554s) [noise=true , address=77.90.31.21:4893 , piece_id=C4E5JVOZ4Z5MBCYVCBCWBNMYWDMFPURSWCGXUXUIMKY3R7ZU4ELA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364723747 (589h17m45.235806782s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364814078 (589h17m45.235897123s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.383057617 (589h17m45.254140662s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.408958516 (589h17m45.280041551s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.413279004 (589h17m45.284362049s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467102871 (589h17m45.338185906s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467128300 (589h17m45.338211335s) [seg_pos={0 0}, upload=516588]: Segment reached optimal threshold of 14 pieces. Dec 30 04:46:45.467342818 (589h17m45.338425863s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467352838 (589h17m45.338435883s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467724515 (589h17m45.33880756s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467733774 (589h17m45.338816829s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467897455 (589h17m45.33898049s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection Dec 30 04:46:45.467905209 (589h17m45.338988254s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468253593 (589h17m45.339336638s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: read tcp 188.166.225.170:56378->77.90.31.11:4588: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:46:45.468263031 (589h17m45.339346076s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468282408 (589h17m45.339365453s) [seg_pos={0 0}, upload=516588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:46:45.468368562 (589h17m45.339451607s) [upload=516588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:46:45.468374774 (589h17m45.339457819s) [upload=516588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:46:45.755910376 (589h17m45.626993421s) [upload=516588]: Done uploading segments... Dec 30 04:46:45.755950452 (589h17m45.627033497s) [upload=516588]: Upload finished. err: <nil> Duration: 15.392092518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.XX..X.....X..............X................X.............X..] duration: 5.000928743s trace: https://jaeger.dev.storj.io/trace/77f156dfc046b501 FAIL (2025-12-30 04:48:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001003172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..................................................X.....X.] duration: 44.19484907s trace: https://jaeger.dev.storj.io/trace/5a71d1d7e99c0ec6 FAIL (2025-12-30 04:48:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.576312321 (589h17m42.447395366s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.577549918 (589h17m42.448632953s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.578974851 (589h17m42.450057886s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.580367783 (589h17m42.451450818s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.581567388 (589h17m42.452650423s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.582814021 (589h17m42.453897056s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.584002213 (589h17m42.455085258s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.585288813 (589h17m42.456371858s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.586586455 (589h17m42.4576695s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.588133640 (589h17m42.459216675s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.589628978 (589h17m42.460712013s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591216792 (589h17m42.462299827s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591966859 (589h17m42.463049894s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593499888 (589h17m42.464582924s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593996395 (589h17m42.46507944s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:44.048371258 (589h17m43.919454293s) [upload=516588]: Next returned nil segment Dec 30 04:46:44.048384013 (589h17m43.919467058s) [upload=516588]: Waiting for error group managing segments... Dec 30 04:46:44.447700633 (589h17m44.318783678s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.453673536 (589h17m44.324756572s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.458257975 (589h17m44.32934101s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.463944624 (589h17m44.335027659s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.469691828 (589h17m44.340774863s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.688602448 (589h17m44.559685493s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.048959672 (589h17m44.920042718s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.199176519 (589h17m45.070259554s) [noise=true , address=77.90.31.21:4893 , piece_id=C4E5JVOZ4Z5MBCYVCBCWBNMYWDMFPURSWCGXUXUIMKY3R7ZU4ELA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364723747 (589h17m45.235806782s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364814078 (589h17m45.235897123s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.383057617 (589h17m45.254140662s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.408958516 (589h17m45.280041551s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.413279004 (589h17m45.284362049s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467102871 (589h17m45.338185906s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467128300 (589h17m45.338211335s) [seg_pos={0 0}, upload=516588]: Segment reached optimal threshold of 14 pieces. Dec 30 04:46:45.467342818 (589h17m45.338425863s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467352838 (589h17m45.338435883s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467724515 (589h17m45.33880756s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467733774 (589h17m45.338816829s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467897455 (589h17m45.33898049s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection Dec 30 04:46:45.467905209 (589h17m45.338988254s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468253593 (589h17m45.339336638s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: read tcp 188.166.225.170:56378->77.90.31.11:4588: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:46:45.468263031 (589h17m45.339346076s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468282408 (589h17m45.339365453s) [seg_pos={0 0}, upload=516588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:46:45.468368562 (589h17m45.339451607s) [upload=516588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:46:45.468374774 (589h17m45.339457819s) [upload=516588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:46:45.755910376 (589h17m45.626993421s) [upload=516588]: Done uploading segments... Dec 30 04:46:45.755950452 (589h17m45.627033497s) [upload=516588]: Upload finished. err: <nil> Duration: 44.194865431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..................................................X.....X] duration: 1m0.001999038s trace: https://jaeger.dev.storj.io/trace/5a71d1d7e99c0ec6 FAIL (2025-12-30 04:47:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.576312321 (589h17m42.447395366s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.577549918 (589h17m42.448632953s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.578974851 (589h17m42.450057886s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.580367783 (589h17m42.451450818s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.581567388 (589h17m42.452650423s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.582814021 (589h17m42.453897056s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.584002213 (589h17m42.455085258s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.585288813 (589h17m42.456371858s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.586586455 (589h17m42.4576695s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.588133640 (589h17m42.459216675s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.589628978 (589h17m42.460712013s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591216792 (589h17m42.462299827s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.591966859 (589h17m42.463049894s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593499888 (589h17m42.464582924s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:42.593996395 (589h17m42.46507944s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Uploading piece... Dec 30 04:46:44.048371258 (589h17m43.919454293s) [upload=516588]: Next returned nil segment Dec 30 04:46:44.048384013 (589h17m43.919467058s) [upload=516588]: Waiting for error group managing segments... Dec 30 04:46:44.447700633 (589h17m44.318783678s) [noise=true , address=145.79.227.34:4918 , piece_id=HEPMFBK4UYWPMVCBSKHSB4I62MX6E6QIZJBSB4DM2XSXSV77JYAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.453673536 (589h17m44.324756572s) [noise=true , address=145.79.227.2:4140 , piece_id=65A7IFRSPQOMQC5HQ4GH72MNNWGWL3KCHWDYEBAFEYAGCVO7WTLA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.458257975 (589h17m44.32934101s) [noise=true , address=145.79.227.35:4496 , piece_id=A5HUNEEXTTMTNVF4TADXNM5JJOOTTELPSPA6OK7LPDXJJJMGVNZA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.463944624 (589h17m44.335027659s) [noise=true , address=145.79.227.33:4875 , piece_id=DGDNWV5EG75XO5FHKUIW3WZ7BAQXWWCXXW53TSZVW7BFWX4EAV4Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.469691828 (589h17m44.340774863s) [noise=true , address=145.79.227.4:4203 , piece_id=XLRPSBFKDSCLTD6CURKOARGG37GCLZYCZHK26BCCHKMXLY5TTOZQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:44.688602448 (589h17m44.559685493s) [noise=true , address=45.91.116.10:4090 , piece_id=O56UC26TK3WJU2G2AZAEQU573K5PU73GKIVP2HOE6GKJQBJVSG3A, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.048959672 (589h17m44.920042718s) [noise=true , address=77.90.31.17:4778 , piece_id=EYE2H5MMSEEARGFRMELJBWS34Z3Q6MRGHVU5W5KIXHKZASVRRCFA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.199176519 (589h17m45.070259554s) [noise=true , address=77.90.31.21:4893 , piece_id=C4E5JVOZ4Z5MBCYVCBCWBNMYWDMFPURSWCGXUXUIMKY3R7ZU4ELA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364723747 (589h17m45.235806782s) [noise=true , address=145.79.227.30:4777 , piece_id=HSPLGCM6PXZXXFXWUVFX63J54HIWWONVZC2MTS4EATMAR66VNVNQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.364814078 (589h17m45.235897123s) [noise=true , address=145.79.227.40:4074 , piece_id=5W7JDRTYJ3IWHG7LQHLOBRDUVCQ2GINQ4OX54AHWZSLYFOPJ6OSQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.383057617 (589h17m45.254140662s) [noise=true , address=145.79.227.29:4973 , piece_id=25PXLAYZXEMALOCVNNDET4IYLPRNYVQDT2YVCPBKQP6XRXHQKO7Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.408958516 (589h17m45.280041551s) [noise=true , address=145.79.227.44:4222 , piece_id=Y3UGT3KENXMVLXOFZRAQEUJKB3HZAYUXPHNCKJ4G6ELUQYG6OD2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.413279004 (589h17m45.284362049s) [noise=true , address=45.91.116.9:4283 , piece_id=ATDLFH7CKUGIR4IWA7XKFZ4RQL3JOP6E5JXKY6B6QS6NSWICZ2LA, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467102871 (589h17m45.338185906s) [noise=true , address=145.79.227.43:4175 , piece_id=EMLOOBTLPDD5YSOUNOTJG7GHEPWJ4NNQVPEEQDOMS72WZITUWRAQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: <nil> Dec 30 04:46:45.467128300 (589h17m45.338211335s) [seg_pos={0 0}, upload=516588]: Segment reached optimal threshold of 14 pieces. Dec 30 04:46:45.467342818 (589h17m45.338425863s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467352838 (589h17m45.338435883s) [noise=true , address=193.149.164.143:4695 , piece_id=6LLKMC4ESS7KGRIQGVK6LJFJ33FFL4A4FHALLJX7S66UJDBKYPPQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467724515 (589h17m45.33880756s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 30 04:46:45.467733774 (589h17m45.338816829s) [noise=true , address=45.91.116.12:4231 , piece_id=WHTRW6XCVJD7QGQFKTLKPHLBLNF2TRMUBFGTILYWIOJ6O4NEQYEQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.467897455 (589h17m45.33898049s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33212->64.49.13.169:4708: use of closed network connection Dec 30 04:46:45.467905209 (589h17m45.338988254s) [noise=true , address=64.49.13.169:4708 , piece_id=6EP52MEPGITGVUYNL4HOORBWO6AYKIXTQMGAXVZHCWFAFDJRUNTQ, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468253593 (589h17m45.339336638s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: read tcp 188.166.225.170:56378->77.90.31.11:4588: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:46:45.468263031 (589h17m45.339346076s) [noise=true , address=77.90.31.11:4588 , piece_id=TIREFEAOEDY4I4VYQJI6VEHKVWGROZ3FNNRDAIY6YJPLJLUO3Z2Q, seg_pos={0 0}, upload=516588]: Long tail cancellation due to optimal threshold Dec 30 04:46:45.468282408 (589h17m45.339365453s) [seg_pos={0 0}, upload=516588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:46:45.468368562 (589h17m45.339451607s) [upload=516588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:46:45.468374774 (589h17m45.339457819s) [upload=516588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:46:45.755910376 (589h17m45.626993421s) [upload=516588]: Done uploading segments... Dec 30 04:46:45.755950452 (589h17m45.627033497s) [upload=516588]: Upload finished. err: <nil> Duration: 1m0.00200971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..XX..X.....X..............X................X.............X.] duration: 3.971357466s trace: https://jaeger.dev.storj.io/trace/77f156dfc046b501 FAIL (2025-12-30 04:47:31 UTC) orbiter warning: context deadline exceeded Duration: 3.971412381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X.....] duration: 6.068631936s trace: https://jaeger.dev.storj.io/trace/22d233046ce52644 FAIL (2025-12-30 04:47:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ) context deadline exceeded ======LOGS====== Dec 30 04:44:00.759372957 (589h15m0.630456002s) [upload=516551]: Starting upload Dec 30 04:44:00.759444684 (589h15m0.630527729s) [upload=516551]: Done starting upload Dec 30 04:44:00.759613736 (589h15m0.630696781s) [upload=516551]: Uploading segments... Dec 30 04:44:00.759713336 (589h15m0.630796371s) [upload=516551]: Got next segment. Inline: false Dec 30 04:44:00.759770705 (589h15m0.63085375s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:44:00.759777478 (589h15m0.630860523s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:44:01.011437116 (589h15m0.882520171s) [seg_pos={0 0}, upload=516551]: Begin upload segment... Dec 30 04:44:01.011483385 (589h15m0.88256642s) [seg_pos={0 0}, upload=516551]: Done begin upload segment. Dec 30 04:44:01.011516878 (589h15m0.882599924s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011547677 (589h15m0.882630722s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011960754 (589h15m0.883043789s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012343563 (589h15m0.883426608s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012715280 (589h15m0.883798325s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013094933 (589h15m0.884177968s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013401757 (589h15m0.884484803s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013713621 (589h15m0.884796656s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014029964 (589h15m0.885113009s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014607444 (589h15m0.885690489s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015090913 (589h15m0.886173958s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015494172 (589h15m0.886577217s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015977352 (589h15m0.887060387s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016701291 (589h15m0.887784336s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016952328 (589h15m0.888035363s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.017890525 (589h15m0.88897357s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:02.635631422 (589h15m2.506714468s) [upload=516551]: Next returned nil segment Dec 30 04:44:02.635645500 (589h15m2.506728545s) [upload=516551]: Waiting for error group managing segments... Dec 30 04:44:03.017939346 (589h15m2.889022381s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.018493411 (589h15m2.889576456s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.020221662 (589h15m2.891304707s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021047605 (589h15m2.89213065s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021991722 (589h15m2.893074767s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.022043071 (589h15m2.893126116s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.023048245 (589h15m2.89413129s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.024957160 (589h15m2.896040195s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.025506715 (589h15m2.896589751s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.026849543 (589h15m2.897932588s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.240848379 (589h15m3.111931424s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.528632795 (589h15m3.39971584s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564326161 (589h15m3.435409206s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564354425 (589h15m3.43543747s) [seg_pos={0 0}, upload=516551]: Segment reached optimal threshold of 13 pieces. Dec 30 04:44:03.564620571 (589h15m3.435703646s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection Dec 30 04:44:03.564634949 (589h15m3.435717994s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564771720 (589h15m3.435854775s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564783662 (589h15m3.435866697s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564932385 (589h15m3.43601543s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564943818 (589h15m3.436026863s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564969586 (589h15m3.436052631s) [seg_pos={0 0}, upload=516551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:44:03.565075447 (589h15m3.436158492s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:44:03.565083263 (589h15m3.436166308s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:44:03.852073766 (589h15m3.723156811s) [upload=516551]: Done uploading segments... Dec 30 04:44:03.852099425 (589h15m3.72318246s) [upload=516551]: Upload finished. err: <nil> Duration: 6.068650224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..................................................X.....] duration: 31.788040383s trace: https://jaeger.dev.storj.io/trace/72f266853017e859 FAIL (2025-12-30 04:46:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918105148 (589h12m58.789188193s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918816583 (589h12m58.789899628s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918855968 (589h12m58.789939013s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919168112 (589h12m58.790251147s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919392008 (589h12m58.790475073s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919460218 (589h12m58.790543253s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919892150 (589h12m58.790975195s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920417591 (589h12m58.791500636s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920429804 (589h12m58.79151285s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920940947 (589h12m58.792023992s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921004228 (589h12m58.792087263s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921398579 (589h12m58.792481624s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921783332 (589h12m58.792866367s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.922187882 (589h12m58.793270917s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:42:00.276402715 (589h13m0.14748576s) [upload=516516]: Next returned nil segment Dec 30 04:42:00.276416642 (589h13m0.147499677s) [upload=516516]: Waiting for error group managing segments... Dec 30 04:42:00.622019383 (589h13m0.493102418s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.834416780 (589h13m0.705499826s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.882545044 (589h13m0.75362809s) [noise=true , address=145.79.227.33:4878 , piece_id=5NKBVEG32CN5UCVEQAR6SZL5X7ZJAHFWF2YNEXSAYGPV2S5JF6RQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.933380432 (589h13m0.804463478s) [noise=true , address=145.79.227.29:4945 , piece_id=72IYCTTNHB323SNQLDTFW36MY6ALS6DB4F6K2DBUTS2RIG27URTA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.947248409 (589h13m0.818331454s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.955957202 (589h13m0.827040237s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.993401712 (589h13m0.864484777s) [noise=true , address=145.79.227.49:4362 , piece_id=P6OVQX2RAO36LGL2YHXWCIJ3YH263IYSK3OP7Q7PBIVFXDJPJUEQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.070384730 (589h13m0.941467775s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.490289353 (589h13m1.361372398s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.624017151 (589h13m1.495100196s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.625117205 (589h13m1.49620025s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.665407685 (589h13m1.53649075s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.687750499 (589h13m1.558833554s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689061385 (589h13m1.56014443s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689077416 (589h13m1.560160461s) [seg_pos={0 0}, upload=516516]: Segment reached optimal threshold of 14 pieces. Dec 30 04:42:01.689310499 (589h13m1.560393555s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection Dec 30 04:42:01.689319697 (589h13m1.560402742s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689470444 (589h13m1.56055352s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection Dec 30 04:42:01.689519187 (589h13m1.560602232s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689634196 (589h13m1.560717252s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: read tcp 188.166.225.170:35826->45.91.116.10:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:42:01.689644616 (589h13m1.560727661s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689927735 (589h13m1.56101077s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:01.689936862 (589h13m1.561019897s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.690061630 (589h13m1.561144675s) [seg_pos={0 0}, upload=516516]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:42:01.690237415 (589h13m1.56132046s) [upload=516516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:42:01.690246662 (589h13m1.561329707s) [upload=516516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:42:01.980465043 (589h13m1.851548088s) [upload=516516]: Done uploading segments... Dec 30 04:42:01.980496303 (589h13m1.851579348s) [upload=516516]: Upload finished. err: <nil> Duration: 31.788064649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...XX..X.....X..............X................X.............X] duration: 5.001452166s trace: https://jaeger.dev.storj.io/trace/77f156dfc046b501 FAIL (2025-12-30 04:46:27 UTC) orbiter warning: context deadline exceeded Duration: 5.001473836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X....] duration: 6.031812592s trace: https://jaeger.dev.storj.io/trace/22d233046ce52644 FAIL (2025-12-30 04:46:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ) context deadline exceeded ======LOGS====== Dec 30 04:44:00.759372957 (589h15m0.630456002s) [upload=516551]: Starting upload Dec 30 04:44:00.759444684 (589h15m0.630527729s) [upload=516551]: Done starting upload Dec 30 04:44:00.759613736 (589h15m0.630696781s) [upload=516551]: Uploading segments... Dec 30 04:44:00.759713336 (589h15m0.630796371s) [upload=516551]: Got next segment. Inline: false Dec 30 04:44:00.759770705 (589h15m0.63085375s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:44:00.759777478 (589h15m0.630860523s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:44:01.011437116 (589h15m0.882520171s) [seg_pos={0 0}, upload=516551]: Begin upload segment... Dec 30 04:44:01.011483385 (589h15m0.88256642s) [seg_pos={0 0}, upload=516551]: Done begin upload segment. Dec 30 04:44:01.011516878 (589h15m0.882599924s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011547677 (589h15m0.882630722s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011960754 (589h15m0.883043789s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012343563 (589h15m0.883426608s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012715280 (589h15m0.883798325s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013094933 (589h15m0.884177968s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013401757 (589h15m0.884484803s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013713621 (589h15m0.884796656s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014029964 (589h15m0.885113009s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014607444 (589h15m0.885690489s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015090913 (589h15m0.886173958s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015494172 (589h15m0.886577217s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015977352 (589h15m0.887060387s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016701291 (589h15m0.887784336s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016952328 (589h15m0.888035363s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.017890525 (589h15m0.88897357s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:02.635631422 (589h15m2.506714468s) [upload=516551]: Next returned nil segment Dec 30 04:44:02.635645500 (589h15m2.506728545s) [upload=516551]: Waiting for error group managing segments... Dec 30 04:44:03.017939346 (589h15m2.889022381s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.018493411 (589h15m2.889576456s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.020221662 (589h15m2.891304707s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021047605 (589h15m2.89213065s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021991722 (589h15m2.893074767s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.022043071 (589h15m2.893126116s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.023048245 (589h15m2.89413129s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.024957160 (589h15m2.896040195s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.025506715 (589h15m2.896589751s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.026849543 (589h15m2.897932588s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.240848379 (589h15m3.111931424s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.528632795 (589h15m3.39971584s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564326161 (589h15m3.435409206s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564354425 (589h15m3.43543747s) [seg_pos={0 0}, upload=516551]: Segment reached optimal threshold of 13 pieces. Dec 30 04:44:03.564620571 (589h15m3.435703646s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection Dec 30 04:44:03.564634949 (589h15m3.435717994s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564771720 (589h15m3.435854775s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564783662 (589h15m3.435866697s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564932385 (589h15m3.43601543s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564943818 (589h15m3.436026863s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564969586 (589h15m3.436052631s) [seg_pos={0 0}, upload=516551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:44:03.565075447 (589h15m3.436158492s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:44:03.565083263 (589h15m3.436166308s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:44:03.852073766 (589h15m3.723156811s) [upload=516551]: Done uploading segments... Dec 30 04:44:03.852099425 (589h15m3.72318246s) [upload=516551]: Upload finished. err: <nil> Duration: 6.031853158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X...] duration: 5.794421889s trace: https://jaeger.dev.storj.io/trace/22d233046ce52644 FAIL (2025-12-30 04:46:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ) context deadline exceeded ======LOGS====== Dec 30 04:44:00.759372957 (589h15m0.630456002s) [upload=516551]: Starting upload Dec 30 04:44:00.759444684 (589h15m0.630527729s) [upload=516551]: Done starting upload Dec 30 04:44:00.759613736 (589h15m0.630696781s) [upload=516551]: Uploading segments... Dec 30 04:44:00.759713336 (589h15m0.630796371s) [upload=516551]: Got next segment. Inline: false Dec 30 04:44:00.759770705 (589h15m0.63085375s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:44:00.759777478 (589h15m0.630860523s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:44:01.011437116 (589h15m0.882520171s) [seg_pos={0 0}, upload=516551]: Begin upload segment... Dec 30 04:44:01.011483385 (589h15m0.88256642s) [seg_pos={0 0}, upload=516551]: Done begin upload segment. Dec 30 04:44:01.011516878 (589h15m0.882599924s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011547677 (589h15m0.882630722s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011960754 (589h15m0.883043789s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012343563 (589h15m0.883426608s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012715280 (589h15m0.883798325s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013094933 (589h15m0.884177968s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013401757 (589h15m0.884484803s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013713621 (589h15m0.884796656s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014029964 (589h15m0.885113009s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014607444 (589h15m0.885690489s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015090913 (589h15m0.886173958s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015494172 (589h15m0.886577217s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015977352 (589h15m0.887060387s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016701291 (589h15m0.887784336s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016952328 (589h15m0.888035363s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.017890525 (589h15m0.88897357s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:02.635631422 (589h15m2.506714468s) [upload=516551]: Next returned nil segment Dec 30 04:44:02.635645500 (589h15m2.506728545s) [upload=516551]: Waiting for error group managing segments... Dec 30 04:44:03.017939346 (589h15m2.889022381s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.018493411 (589h15m2.889576456s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.020221662 (589h15m2.891304707s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021047605 (589h15m2.89213065s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021991722 (589h15m2.893074767s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.022043071 (589h15m2.893126116s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.023048245 (589h15m2.89413129s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.024957160 (589h15m2.896040195s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.025506715 (589h15m2.896589751s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.026849543 (589h15m2.897932588s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.240848379 (589h15m3.111931424s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.528632795 (589h15m3.39971584s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564326161 (589h15m3.435409206s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564354425 (589h15m3.43543747s) [seg_pos={0 0}, upload=516551]: Segment reached optimal threshold of 13 pieces. Dec 30 04:44:03.564620571 (589h15m3.435703646s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection Dec 30 04:44:03.564634949 (589h15m3.435717994s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564771720 (589h15m3.435854775s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564783662 (589h15m3.435866697s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564932385 (589h15m3.43601543s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564943818 (589h15m3.436026863s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564969586 (589h15m3.436052631s) [seg_pos={0 0}, upload=516551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:44:03.565075447 (589h15m3.436158492s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:44:03.565083263 (589h15m3.436166308s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:44:03.852073766 (589h15m3.723156811s) [upload=516551]: Done uploading segments... Dec 30 04:44:03.852099425 (589h15m3.72318246s) [upload=516551]: Upload finished. err: <nil> Duration: 5.79444812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..................................................X....] duration: 44.453050525s trace: https://jaeger.dev.storj.io/trace/72f266853017e859 FAIL (2025-12-30 04:45:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918105148 (589h12m58.789188193s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918816583 (589h12m58.789899628s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918855968 (589h12m58.789939013s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919168112 (589h12m58.790251147s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919392008 (589h12m58.790475073s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919460218 (589h12m58.790543253s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919892150 (589h12m58.790975195s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920417591 (589h12m58.791500636s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920429804 (589h12m58.79151285s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920940947 (589h12m58.792023992s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921004228 (589h12m58.792087263s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921398579 (589h12m58.792481624s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921783332 (589h12m58.792866367s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.922187882 (589h12m58.793270917s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:42:00.276402715 (589h13m0.14748576s) [upload=516516]: Next returned nil segment Dec 30 04:42:00.276416642 (589h13m0.147499677s) [upload=516516]: Waiting for error group managing segments... Dec 30 04:42:00.622019383 (589h13m0.493102418s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.834416780 (589h13m0.705499826s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.882545044 (589h13m0.75362809s) [noise=true , address=145.79.227.33:4878 , piece_id=5NKBVEG32CN5UCVEQAR6SZL5X7ZJAHFWF2YNEXSAYGPV2S5JF6RQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.933380432 (589h13m0.804463478s) [noise=true , address=145.79.227.29:4945 , piece_id=72IYCTTNHB323SNQLDTFW36MY6ALS6DB4F6K2DBUTS2RIG27URTA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.947248409 (589h13m0.818331454s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.955957202 (589h13m0.827040237s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.993401712 (589h13m0.864484777s) [noise=true , address=145.79.227.49:4362 , piece_id=P6OVQX2RAO36LGL2YHXWCIJ3YH263IYSK3OP7Q7PBIVFXDJPJUEQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.070384730 (589h13m0.941467775s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.490289353 (589h13m1.361372398s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.624017151 (589h13m1.495100196s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.625117205 (589h13m1.49620025s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.665407685 (589h13m1.53649075s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.687750499 (589h13m1.558833554s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689061385 (589h13m1.56014443s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689077416 (589h13m1.560160461s) [seg_pos={0 0}, upload=516516]: Segment reached optimal threshold of 14 pieces. Dec 30 04:42:01.689310499 (589h13m1.560393555s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection Dec 30 04:42:01.689319697 (589h13m1.560402742s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689470444 (589h13m1.56055352s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection Dec 30 04:42:01.689519187 (589h13m1.560602232s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689634196 (589h13m1.560717252s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: read tcp 188.166.225.170:35826->45.91.116.10:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:42:01.689644616 (589h13m1.560727661s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689927735 (589h13m1.56101077s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:01.689936862 (589h13m1.561019897s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.690061630 (589h13m1.561144675s) [seg_pos={0 0}, upload=516516]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:42:01.690237415 (589h13m1.56132046s) [upload=516516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:42:01.690246662 (589h13m1.561329707s) [upload=516516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:42:01.980465043 (589h13m1.851548088s) [upload=516516]: Done uploading segments... Dec 30 04:42:01.980496303 (589h13m1.851579348s) [upload=516516]: Upload finished. err: <nil> Duration: 44.453066966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....XX..X.....X..............X................X.............] duration: 3.433915624s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:45:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.433944418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X..] duration: 5.899921391s trace: https://jaeger.dev.storj.io/trace/22d233046ce52644 FAIL (2025-12-30 04:44:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ) context deadline exceeded ======LOGS====== Dec 30 04:44:00.759372957 (589h15m0.630456002s) [upload=516551]: Starting upload Dec 30 04:44:00.759444684 (589h15m0.630527729s) [upload=516551]: Done starting upload Dec 30 04:44:00.759613736 (589h15m0.630696781s) [upload=516551]: Uploading segments... Dec 30 04:44:00.759713336 (589h15m0.630796371s) [upload=516551]: Got next segment. Inline: false Dec 30 04:44:00.759770705 (589h15m0.63085375s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:44:00.759777478 (589h15m0.630860523s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:44:01.011437116 (589h15m0.882520171s) [seg_pos={0 0}, upload=516551]: Begin upload segment... Dec 30 04:44:01.011483385 (589h15m0.88256642s) [seg_pos={0 0}, upload=516551]: Done begin upload segment. Dec 30 04:44:01.011516878 (589h15m0.882599924s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011547677 (589h15m0.882630722s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011960754 (589h15m0.883043789s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012343563 (589h15m0.883426608s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012715280 (589h15m0.883798325s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013094933 (589h15m0.884177968s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013401757 (589h15m0.884484803s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013713621 (589h15m0.884796656s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014029964 (589h15m0.885113009s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014607444 (589h15m0.885690489s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015090913 (589h15m0.886173958s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015494172 (589h15m0.886577217s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015977352 (589h15m0.887060387s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016701291 (589h15m0.887784336s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016952328 (589h15m0.888035363s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.017890525 (589h15m0.88897357s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:02.635631422 (589h15m2.506714468s) [upload=516551]: Next returned nil segment Dec 30 04:44:02.635645500 (589h15m2.506728545s) [upload=516551]: Waiting for error group managing segments... Dec 30 04:44:03.017939346 (589h15m2.889022381s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.018493411 (589h15m2.889576456s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.020221662 (589h15m2.891304707s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021047605 (589h15m2.89213065s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021991722 (589h15m2.893074767s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.022043071 (589h15m2.893126116s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.023048245 (589h15m2.89413129s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.024957160 (589h15m2.896040195s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.025506715 (589h15m2.896589751s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.026849543 (589h15m2.897932588s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.240848379 (589h15m3.111931424s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.528632795 (589h15m3.39971584s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564326161 (589h15m3.435409206s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564354425 (589h15m3.43543747s) [seg_pos={0 0}, upload=516551]: Segment reached optimal threshold of 13 pieces. Dec 30 04:44:03.564620571 (589h15m3.435703646s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection Dec 30 04:44:03.564634949 (589h15m3.435717994s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564771720 (589h15m3.435854775s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564783662 (589h15m3.435866697s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564932385 (589h15m3.43601543s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564943818 (589h15m3.436026863s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564969586 (589h15m3.436052631s) [seg_pos={0 0}, upload=516551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:44:03.565075447 (589h15m3.436158492s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:44:03.565083263 (589h15m3.436166308s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:44:03.852073766 (589h15m3.723156811s) [upload=516551]: Done uploading segments... Dec 30 04:44:03.852099425 (589h15m3.72318246s) [upload=516551]: Upload finished. err: <nil> Duration: 5.899942139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..................................................X...] duration: 36.376148537s trace: https://jaeger.dev.storj.io/trace/72f266853017e859 FAIL (2025-12-30 04:44:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918105148 (589h12m58.789188193s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918816583 (589h12m58.789899628s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918855968 (589h12m58.789939013s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919168112 (589h12m58.790251147s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919392008 (589h12m58.790475073s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919460218 (589h12m58.790543253s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919892150 (589h12m58.790975195s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920417591 (589h12m58.791500636s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920429804 (589h12m58.79151285s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920940947 (589h12m58.792023992s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921004228 (589h12m58.792087263s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921398579 (589h12m58.792481624s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921783332 (589h12m58.792866367s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.922187882 (589h12m58.793270917s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:42:00.276402715 (589h13m0.14748576s) [upload=516516]: Next returned nil segment Dec 30 04:42:00.276416642 (589h13m0.147499677s) [upload=516516]: Waiting for error group managing segments... Dec 30 04:42:00.622019383 (589h13m0.493102418s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.834416780 (589h13m0.705499826s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.882545044 (589h13m0.75362809s) [noise=true , address=145.79.227.33:4878 , piece_id=5NKBVEG32CN5UCVEQAR6SZL5X7ZJAHFWF2YNEXSAYGPV2S5JF6RQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.933380432 (589h13m0.804463478s) [noise=true , address=145.79.227.29:4945 , piece_id=72IYCTTNHB323SNQLDTFW36MY6ALS6DB4F6K2DBUTS2RIG27URTA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.947248409 (589h13m0.818331454s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.955957202 (589h13m0.827040237s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.993401712 (589h13m0.864484777s) [noise=true , address=145.79.227.49:4362 , piece_id=P6OVQX2RAO36LGL2YHXWCIJ3YH263IYSK3OP7Q7PBIVFXDJPJUEQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.070384730 (589h13m0.941467775s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.490289353 (589h13m1.361372398s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.624017151 (589h13m1.495100196s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.625117205 (589h13m1.49620025s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.665407685 (589h13m1.53649075s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.687750499 (589h13m1.558833554s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689061385 (589h13m1.56014443s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689077416 (589h13m1.560160461s) [seg_pos={0 0}, upload=516516]: Segment reached optimal threshold of 14 pieces. Dec 30 04:42:01.689310499 (589h13m1.560393555s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection Dec 30 04:42:01.689319697 (589h13m1.560402742s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689470444 (589h13m1.56055352s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection Dec 30 04:42:01.689519187 (589h13m1.560602232s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689634196 (589h13m1.560717252s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: read tcp 188.166.225.170:35826->45.91.116.10:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:42:01.689644616 (589h13m1.560727661s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689927735 (589h13m1.56101077s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:01.689936862 (589h13m1.561019897s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.690061630 (589h13m1.561144675s) [seg_pos={0 0}, upload=516516]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:42:01.690237415 (589h13m1.56132046s) [upload=516516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:42:01.690246662 (589h13m1.561329707s) [upload=516516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:42:01.980465043 (589h13m1.851548088s) [upload=516516]: Done uploading segments... Dec 30 04:42:01.980496303 (589h13m1.851579348s) [upload=516516]: Upload finished. err: <nil> Duration: 36.376179818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X.] duration: 5.769708967s trace: https://jaeger.dev.storj.io/trace/22d233046ce52644 FAIL (2025-12-30 04:44:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ) context deadline exceeded ======LOGS====== Dec 30 04:44:00.759372957 (589h15m0.630456002s) [upload=516551]: Starting upload Dec 30 04:44:00.759444684 (589h15m0.630527729s) [upload=516551]: Done starting upload Dec 30 04:44:00.759613736 (589h15m0.630696781s) [upload=516551]: Uploading segments... Dec 30 04:44:00.759713336 (589h15m0.630796371s) [upload=516551]: Got next segment. Inline: false Dec 30 04:44:00.759770705 (589h15m0.63085375s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:44:00.759777478 (589h15m0.630860523s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:44:01.011437116 (589h15m0.882520171s) [seg_pos={0 0}, upload=516551]: Begin upload segment... Dec 30 04:44:01.011483385 (589h15m0.88256642s) [seg_pos={0 0}, upload=516551]: Done begin upload segment. Dec 30 04:44:01.011516878 (589h15m0.882599924s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011547677 (589h15m0.882630722s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011960754 (589h15m0.883043789s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012343563 (589h15m0.883426608s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012715280 (589h15m0.883798325s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013094933 (589h15m0.884177968s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013401757 (589h15m0.884484803s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013713621 (589h15m0.884796656s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014029964 (589h15m0.885113009s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014607444 (589h15m0.885690489s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015090913 (589h15m0.886173958s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015494172 (589h15m0.886577217s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015977352 (589h15m0.887060387s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016701291 (589h15m0.887784336s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016952328 (589h15m0.888035363s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.017890525 (589h15m0.88897357s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:02.635631422 (589h15m2.506714468s) [upload=516551]: Next returned nil segment Dec 30 04:44:02.635645500 (589h15m2.506728545s) [upload=516551]: Waiting for error group managing segments... Dec 30 04:44:03.017939346 (589h15m2.889022381s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.018493411 (589h15m2.889576456s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.020221662 (589h15m2.891304707s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021047605 (589h15m2.89213065s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021991722 (589h15m2.893074767s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.022043071 (589h15m2.893126116s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.023048245 (589h15m2.89413129s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.024957160 (589h15m2.896040195s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.025506715 (589h15m2.896589751s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.026849543 (589h15m2.897932588s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.240848379 (589h15m3.111931424s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.528632795 (589h15m3.39971584s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564326161 (589h15m3.435409206s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564354425 (589h15m3.43543747s) [seg_pos={0 0}, upload=516551]: Segment reached optimal threshold of 13 pieces. Dec 30 04:44:03.564620571 (589h15m3.435703646s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection Dec 30 04:44:03.564634949 (589h15m3.435717994s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564771720 (589h15m3.435854775s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564783662 (589h15m3.435866697s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564932385 (589h15m3.43601543s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564943818 (589h15m3.436026863s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564969586 (589h15m3.436052631s) [seg_pos={0 0}, upload=516551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:44:03.565075447 (589h15m3.436158492s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:44:03.565083263 (589h15m3.436166308s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:44:03.852073766 (589h15m3.723156811s) [upload=516551]: Done uploading segments... Dec 30 04:44:03.852099425 (589h15m3.72318246s) [upload=516551]: Upload finished. err: <nil> Duration: 5.769734244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.X] duration: 30.002676405s trace: https://jaeger.dev.storj.io/trace/22d233046ce52644 FAIL (2025-12-30 04:44:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ) context deadline exceeded ======LOGS====== Dec 30 04:44:00.759372957 (589h15m0.630456002s) [upload=516551]: Starting upload Dec 30 04:44:00.759444684 (589h15m0.630527729s) [upload=516551]: Done starting upload Dec 30 04:44:00.759613736 (589h15m0.630696781s) [upload=516551]: Uploading segments... Dec 30 04:44:00.759713336 (589h15m0.630796371s) [upload=516551]: Got next segment. Inline: false Dec 30 04:44:00.759770705 (589h15m0.63085375s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:44:00.759777478 (589h15m0.630860523s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:44:01.011437116 (589h15m0.882520171s) [seg_pos={0 0}, upload=516551]: Begin upload segment... Dec 30 04:44:01.011483385 (589h15m0.88256642s) [seg_pos={0 0}, upload=516551]: Done begin upload segment. Dec 30 04:44:01.011516878 (589h15m0.882599924s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011547677 (589h15m0.882630722s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.011960754 (589h15m0.883043789s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012343563 (589h15m0.883426608s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.012715280 (589h15m0.883798325s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013094933 (589h15m0.884177968s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013401757 (589h15m0.884484803s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.013713621 (589h15m0.884796656s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014029964 (589h15m0.885113009s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.014607444 (589h15m0.885690489s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015090913 (589h15m0.886173958s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015494172 (589h15m0.886577217s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.015977352 (589h15m0.887060387s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016701291 (589h15m0.887784336s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.016952328 (589h15m0.888035363s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:01.017890525 (589h15m0.88897357s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Uploading piece... Dec 30 04:44:02.635631422 (589h15m2.506714468s) [upload=516551]: Next returned nil segment Dec 30 04:44:02.635645500 (589h15m2.506728545s) [upload=516551]: Waiting for error group managing segments... Dec 30 04:44:03.017939346 (589h15m2.889022381s) [noise=true , address=145.79.227.32:4836 , piece_id=D6CFIOVLLVAAP6CXI7OFO5VDBVNS7P6VMXDK2VUDKV2X6GNFH6JQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.018493411 (589h15m2.889576456s) [noise=true , address=145.79.227.50:4375 , piece_id=JTAW2IJHQOWMNFFCMHMBNCPHGTMHPWKRUGO35N6VT7NA4OXEPENQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.020221662 (589h15m2.891304707s) [noise=true , address=145.79.227.15:4875 , piece_id=EIWS3M5O37F2UTXZLDYUDIN36EWSKFUTGGUDNKO4L4W26CYXC2PQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021047605 (589h15m2.89213065s) [noise=true , address=145.79.227.10:4402 , piece_id=2D26GBJVXDSKCZK7JNJOL7UVW4LTBCOFEBAK462EXEZ2XAFDGIYQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.021991722 (589h15m2.893074767s) [noise=true , address=145.79.227.41:4087 , piece_id=R2RMVBV5Y5DQAE6TBR4P2W2WWOAJK6XDUDR6SDWYMX7KMS3WVXZQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.022043071 (589h15m2.893126116s) [noise=true , address=145.79.227.4:4185 , piece_id=6NGUQ7BFNA6HTFIMCIMASI3WDHADAR2ZDMVCHGDTZZGIANFDPU5Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.023048245 (589h15m2.89413129s) [noise=true , address=145.79.227.7:4294 , piece_id=ILNVXMWFVNF7AY4ILLYOXJI5QAQDWUHF72X6KTENZITCDGZPR4RQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.024957160 (589h15m2.896040195s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOABQSEQTEMLL7DPVFT5OKO5HRYHI66DTEAVDNAWYUZUHLPIE3Q, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.025506715 (589h15m2.896589751s) [noise=true , address=145.79.227.28:4727 , piece_id=QCT4TF6SN3D7AXLBRIOBFWPOJO63EACROXL2BGPDABTE3YOIENIA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.026849543 (589h15m2.897932588s) [noise=true , address=145.79.227.3:4150 , piece_id=CIIZ2ODGM345A3YLB5L3DCDJ5UHIWJPYVEQTENURKJRF6N3J7UPA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.240848379 (589h15m3.111931424s) [noise=true , address=45.91.116.7:4010 , piece_id=VAVCQ7NT4N2UGFERNH6E7NSCSFE6TYRWSP2R3P4DHO55NGTHVV5A, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.528632795 (589h15m3.39971584s) [noise=true , address=77.90.31.14:4663 , piece_id=VI4PC2KG6OLKP3NZOL6T4DIVKKZR6DHN2AWDVOTHDC7JAV7KSQZA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564326161 (589h15m3.435409206s) [noise=true , address=77.90.31.10:4520 , piece_id=OFT63CBP7MZLB3IL25XDU2Y4YPFKY5QKRLY3TKRIV2SIZNCQ65DQ, seg_pos={0 0}, upload=516551]: Done uploading piece. err: <nil> Dec 30 04:44:03.564354425 (589h15m3.43543747s) [seg_pos={0 0}, upload=516551]: Segment reached optimal threshold of 13 pieces. Dec 30 04:44:03.564620571 (589h15m3.435703646s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59270->45.91.116.14:4192: use of closed network connection Dec 30 04:44:03.564634949 (589h15m3.435717994s) [noise=true , address=45.91.116.14:4192 , piece_id=AKULE47EAYRXMVISQ6AD4H42MHHO2EBEXAHBJICF3WBRON3MFKCA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564771720 (589h15m3.435854775s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564783662 (589h15m3.435866697s) [noise=true , address=145.79.227.34:4906 , piece_id=NKOWNREAOFQL5SQJMKL4HNXDNT5PU7KYDGBXFZQMUJ6VGCYFGJRA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564932385 (589h15m3.43601543s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:44:03.564943818 (589h15m3.436026863s) [noise=true , address=45.91.116.4:4940 , piece_id=KUHCVODTAV3UUEQZAKL5GIT2GIJN2MTOU5Z6F7K6YPI7OVA7U5JA, seg_pos={0 0}, upload=516551]: Long tail cancellation due to optimal threshold Dec 30 04:44:03.564969586 (589h15m3.436052631s) [seg_pos={0 0}, upload=516551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:44:03.565075447 (589h15m3.436158492s) [upload=516551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:44:03.565083263 (589h15m3.436166308s) [upload=516551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:44:03.852073766 (589h15m3.723156811s) [upload=516551]: Done uploading segments... Dec 30 04:44:03.852099425 (589h15m3.72318246s) [upload=516551]: Upload finished. err: <nil> Duration: 30.002703426s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....XX..X.....X..............X................X............] duration: 3.450009411s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:44:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45003483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..................................................X..] duration: 6.420348412s trace: https://jaeger.dev.storj.io/trace/72f266853017e859 FAIL (2025-12-30 04:43:51 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918105148 (589h12m58.789188193s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918816583 (589h12m58.789899628s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918855968 (589h12m58.789939013s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919168112 (589h12m58.790251147s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919392008 (589h12m58.790475073s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919460218 (589h12m58.790543253s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919892150 (589h12m58.790975195s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920417591 (589h12m58.791500636s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920429804 (589h12m58.79151285s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920940947 (589h12m58.792023992s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921004228 (589h12m58.792087263s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921398579 (589h12m58.792481624s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921783332 (589h12m58.792866367s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.922187882 (589h12m58.793270917s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:42:00.276402715 (589h13m0.14748576s) [upload=516516]: Next returned nil segment Dec 30 04:42:00.276416642 (589h13m0.147499677s) [upload=516516]: Waiting for error group managing segments... Dec 30 04:42:00.622019383 (589h13m0.493102418s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.834416780 (589h13m0.705499826s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.882545044 (589h13m0.75362809s) [noise=true , address=145.79.227.33:4878 , piece_id=5NKBVEG32CN5UCVEQAR6SZL5X7ZJAHFWF2YNEXSAYGPV2S5JF6RQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.933380432 (589h13m0.804463478s) [noise=true , address=145.79.227.29:4945 , piece_id=72IYCTTNHB323SNQLDTFW36MY6ALS6DB4F6K2DBUTS2RIG27URTA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.947248409 (589h13m0.818331454s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.955957202 (589h13m0.827040237s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.993401712 (589h13m0.864484777s) [noise=true , address=145.79.227.49:4362 , piece_id=P6OVQX2RAO36LGL2YHXWCIJ3YH263IYSK3OP7Q7PBIVFXDJPJUEQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.070384730 (589h13m0.941467775s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.490289353 (589h13m1.361372398s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.624017151 (589h13m1.495100196s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.625117205 (589h13m1.49620025s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.665407685 (589h13m1.53649075s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.687750499 (589h13m1.558833554s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689061385 (589h13m1.56014443s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689077416 (589h13m1.560160461s) [seg_pos={0 0}, upload=516516]: Segment reached optimal threshold of 14 pieces. Dec 30 04:42:01.689310499 (589h13m1.560393555s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection Dec 30 04:42:01.689319697 (589h13m1.560402742s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689470444 (589h13m1.56055352s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection Dec 30 04:42:01.689519187 (589h13m1.560602232s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689634196 (589h13m1.560717252s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: read tcp 188.166.225.170:35826->45.91.116.10:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:42:01.689644616 (589h13m1.560727661s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689927735 (589h13m1.56101077s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:01.689936862 (589h13m1.561019897s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.690061630 (589h13m1.561144675s) [seg_pos={0 0}, upload=516516]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:42:01.690237415 (589h13m1.56132046s) [upload=516516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:42:01.690246662 (589h13m1.561329707s) [upload=516516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:42:01.980465043 (589h13m1.851548088s) [upload=516516]: Done uploading segments... Dec 30 04:42:01.980496303 (589h13m1.851579348s) [upload=516516]: Upload finished. err: <nil> Duration: 6.420365344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..................................................X.] duration: 46.72822901s trace: https://jaeger.dev.storj.io/trace/72f266853017e859 FAIL (2025-12-30 04:43:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918105148 (589h12m58.789188193s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918816583 (589h12m58.789899628s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918855968 (589h12m58.789939013s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919168112 (589h12m58.790251147s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919392008 (589h12m58.790475073s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919460218 (589h12m58.790543253s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919892150 (589h12m58.790975195s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920417591 (589h12m58.791500636s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920429804 (589h12m58.79151285s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920940947 (589h12m58.792023992s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921004228 (589h12m58.792087263s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921398579 (589h12m58.792481624s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921783332 (589h12m58.792866367s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.922187882 (589h12m58.793270917s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:42:00.276402715 (589h13m0.14748576s) [upload=516516]: Next returned nil segment Dec 30 04:42:00.276416642 (589h13m0.147499677s) [upload=516516]: Waiting for error group managing segments... Dec 30 04:42:00.622019383 (589h13m0.493102418s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.834416780 (589h13m0.705499826s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.882545044 (589h13m0.75362809s) [noise=true , address=145.79.227.33:4878 , piece_id=5NKBVEG32CN5UCVEQAR6SZL5X7ZJAHFWF2YNEXSAYGPV2S5JF6RQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.933380432 (589h13m0.804463478s) [noise=true , address=145.79.227.29:4945 , piece_id=72IYCTTNHB323SNQLDTFW36MY6ALS6DB4F6K2DBUTS2RIG27URTA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.947248409 (589h13m0.818331454s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.955957202 (589h13m0.827040237s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.993401712 (589h13m0.864484777s) [noise=true , address=145.79.227.49:4362 , piece_id=P6OVQX2RAO36LGL2YHXWCIJ3YH263IYSK3OP7Q7PBIVFXDJPJUEQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.070384730 (589h13m0.941467775s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.490289353 (589h13m1.361372398s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.624017151 (589h13m1.495100196s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.625117205 (589h13m1.49620025s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.665407685 (589h13m1.53649075s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.687750499 (589h13m1.558833554s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689061385 (589h13m1.56014443s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689077416 (589h13m1.560160461s) [seg_pos={0 0}, upload=516516]: Segment reached optimal threshold of 14 pieces. Dec 30 04:42:01.689310499 (589h13m1.560393555s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection Dec 30 04:42:01.689319697 (589h13m1.560402742s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689470444 (589h13m1.56055352s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection Dec 30 04:42:01.689519187 (589h13m1.560602232s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689634196 (589h13m1.560717252s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: read tcp 188.166.225.170:35826->45.91.116.10:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:42:01.689644616 (589h13m1.560727661s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689927735 (589h13m1.56101077s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:01.689936862 (589h13m1.561019897s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.690061630 (589h13m1.561144675s) [seg_pos={0 0}, upload=516516]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:42:01.690237415 (589h13m1.56132046s) [upload=516516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:42:01.690246662 (589h13m1.561329707s) [upload=516516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:42:01.980465043 (589h13m1.851548088s) [upload=516516]: Done uploading segments... Dec 30 04:42:01.980496303 (589h13m1.851579348s) [upload=516516]: Upload finished. err: <nil> Duration: 46.728251153s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......XX..X.....X..............X................X...........] duration: 3.628578833s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:43:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.628605214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X.] duration: 16.106894299s trace: https://jaeger.dev.storj.io/trace/35f72752c1d7bd4b FAIL (2025-12-30 04:43:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:42:23.754942136 (589h13m23.626025171s) [upload=516526]: Starting upload Dec 30 04:42:23.754997481 (589h13m23.626080516s) [upload=516526]: Done starting upload Dec 30 04:42:23.755191541 (589h13m23.626274576s) [upload=516526]: Uploading segments... Dec 30 04:42:23.755228381 (589h13m23.626311426s) [upload=516526]: Got next segment. Inline: false Dec 30 04:42:23.755253549 (589h13m23.626336584s) [upload=516526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:42:23.755257526 (589h13m23.626340561s) [upload=516526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:42:24.006723045 (589h13m23.87780609s) [seg_pos={0 0}, upload=516526]: Begin upload segment... Dec 30 04:42:24.006794662 (589h13m23.877877707s) [seg_pos={0 0}, upload=516526]: Done begin upload segment. Dec 30 04:42:24.006914370 (589h13m23.877997445s) [noise=true , address=145.79.227.2:4114 , piece_id=PJHNJVO23H4RMMLP53I3GQ3MPW5I54XARNRN32FNUUKY6L5WTNEA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.007678395 (589h13m23.87876143s) [noise=true , address=145.79.227.47:4267 , piece_id=CP3DOZXWAEAQMMBBXCXNBL4WAE7ETPQPHZSJHEVS4X4BFMXNTTCA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.008522763 (589h13m23.879605798s) [noise=true , address=145.79.227.37:4566 , piece_id=NFWG4IZSLYROX7WEKZUDHBULY4HLWB3AVOGITTGYIQNS4MLW4NPA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.008863952 (589h13m23.879947028s) [noise=true , address=45.91.116.8:4046 , piece_id=R473EV2AC7UCHYG5RSQMHFDF2ZIGGSBN3KV7BU6KASVEM43XUAUA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.009090754 (589h13m23.880173799s) [noise=true , address=145.79.227.42:4121 , piece_id=2GSQFKSUI4TQYTMYQTRRYSOGNCZ6NLWOM6KNFU4ARYTW3MGNPTTA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.009340290 (589h13m23.880423335s) [noise=true , address=145.79.227.51:4409 , piece_id=UJC3734NTIPXZ45C4ZFMMJ2KYVA5XEEI4O6UZUZPBGU7K3PIHZAQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.009584024 (589h13m23.880667059s) [noise=true , address=145.79.227.13:4802 , piece_id=ZBFQJPYVVIEBSTAHKBE4M4UVLG6FXMN64MEHSYOIGQFZAAMFIT3A, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.009855710 (589h13m23.880938756s) [noise=true , address=145.79.227.45:4230 , piece_id=4VVGLT4B7LHRFYX7ZK3LSJDCGGSGAPMKJKN3UKATFPCTI463UDNQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.010362376 (589h13m23.881445411s) [noise=true , address=145.79.227.17:4971 , piece_id=UZ3HJKZ445REIZMGZRFC52LPCTDOSE7UFL5JI3A36IKHE2KRP2CA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.011016832 (589h13m23.882099877s) [noise=true , address=77.90.31.14:4663 , piece_id=3IMLBWQJQ5ONGNVYKH2ZANY5CA32T2IRW2B4I2EN3C2X3J7METZA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.011464804 (589h13m23.882547849s) [noise=true , address=45.91.116.12:4225 , piece_id=SYPMZDE4EV26DOB6VETJKF6W3QHPBOWICPOZJTY3MV3RZOTXP3FQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.011894293 (589h13m23.882977348s) [noise=true , address=77.90.31.17:4770 , piece_id=GRIPENYH3JRANVO3IBOVSO7UBQGDAUVN2HGNLZXW5AKPHX53V6WA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.012204063 (589h13m23.883287108s) [noise=true , address=145.79.227.40:4049 , piece_id=7QNC6DTSVAIT3FLGJJUX3III6KOZ7DFTXK73GYNXRWE52IZ2CXPQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.012855583 (589h13m23.883938628s) [noise=true , address=145.79.227.3:4151 , piece_id=NM4MK3FBBIGBXJFWGVDJONV5UZOL4ET4COULEJEXKDR6EJ22QATA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.013297494 (589h13m23.884380539s) [noise=true , address=77.90.31.21:4880 , piece_id=OFZPIKNFMGEORXFBTE63F4QZNWVZSTOODTTOEFQPHXNDZTBKE3DQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.013759574 (589h13m23.884842619s) [noise=true , address=145.79.227.50:4374 , piece_id=P2L5LDFBQFNIHZU3PV7MDAYSAOQSECU6TTFDRLUUJLQOPZ34GJIA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:25.614763512 (589h13m25.485846557s) [upload=516526]: Next returned nil segment Dec 30 04:42:25.614779503 (589h13m25.485862548s) [upload=516526]: Waiting for error group managing segments... Dec 30 04:42:25.994462638 (589h13m25.865545673s) [noise=true , address=145.79.227.2:4114 , piece_id=PJHNJVO23H4RMMLP53I3GQ3MPW5I54XARNRN32FNUUKY6L5WTNEA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:25.999413976 (589h13m25.870497011s) [noise=true , address=145.79.227.50:4374 , piece_id=P2L5LDFBQFNIHZU3PV7MDAYSAOQSECU6TTFDRLUUJLQOPZ34GJIA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:25.999830249 (589h13m25.870913294s) [noise=true , address=145.79.227.45:4230 , piece_id=4VVGLT4B7LHRFYX7ZK3LSJDCGGSGAPMKJKN3UKATFPCTI463UDNQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.004947744 (589h13m25.876030789s) [noise=true , address=145.79.227.47:4267 , piece_id=CP3DOZXWAEAQMMBBXCXNBL4WAE7ETPQPHZSJHEVS4X4BFMXNTTCA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.005538118 (589h13m25.876621153s) [noise=true , address=145.79.227.51:4409 , piece_id=UJC3734NTIPXZ45C4ZFMMJ2KYVA5XEEI4O6UZUZPBGU7K3PIHZAQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.007606777 (589h13m25.878689822s) [noise=true , address=145.79.227.17:4971 , piece_id=UZ3HJKZ445REIZMGZRFC52LPCTDOSE7UFL5JI3A36IKHE2KRP2CA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.015592954 (589h13m25.886676029s) [noise=true , address=145.79.227.3:4151 , piece_id=NM4MK3FBBIGBXJFWGVDJONV5UZOL4ET4COULEJEXKDR6EJ22QATA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.020051263 (589h13m25.891138877s) [noise=true , address=145.79.227.37:4566 , piece_id=NFWG4IZSLYROX7WEKZUDHBULY4HLWB3AVOGITTGYIQNS4MLW4NPA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.021901977 (589h13m25.892985022s) [noise=true , address=145.79.227.13:4802 , piece_id=ZBFQJPYVVIEBSTAHKBE4M4UVLG6FXMN64MEHSYOIGQFZAAMFIT3A, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.038871139 (589h13m25.909954184s) [noise=true , address=145.79.227.42:4121 , piece_id=2GSQFKSUI4TQYTMYQTRRYSOGNCZ6NLWOM6KNFU4ARYTW3MGNPTTA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.185232855 (589h13m26.05631589s) [noise=true , address=77.90.31.17:4770 , piece_id=GRIPENYH3JRANVO3IBOVSO7UBQGDAUVN2HGNLZXW5AKPHX53V6WA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.215356966 (589h13m26.086440001s) [noise=true , address=45.91.116.8:4046 , piece_id=R473EV2AC7UCHYG5RSQMHFDF2ZIGGSBN3KV7BU6KASVEM43XUAUA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.326015306 (589h13m26.197098342s) [noise=true , address=77.90.31.21:4880 , piece_id=OFZPIKNFMGEORXFBTE63F4QZNWVZSTOODTTOEFQPHXNDZTBKE3DQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.326051115 (589h13m26.19713415s) [seg_pos={0 0}, upload=516526]: Segment reached optimal threshold of 13 pieces. Dec 30 04:42:26.326254251 (589h13m26.197337286s) [noise=true , address=145.79.227.40:4049 , piece_id=7QNC6DTSVAIT3FLGJJUX3III6KOZ7DFTXK73GYNXRWE52IZ2CXPQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:26.326261926 (589h13m26.197344961s) [noise=true , address=145.79.227.40:4049 , piece_id=7QNC6DTSVAIT3FLGJJUX3III6KOZ7DFTXK73GYNXRWE52IZ2CXPQ, seg_pos={0 0}, upload=516526]: Long tail cancellation due to optimal threshold Dec 30 04:42:26.326547259 (589h13m26.197630304s) [noise=true , address=45.91.116.12:4225 , piece_id=SYPMZDE4EV26DOB6VETJKF6W3QHPBOWICPOZJTY3MV3RZOTXP3FQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:32938->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:32938->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32938->45.91.116.12:4225: use of closed network connection Dec 30 04:42:26.326566565 (589h13m26.1976496s) [noise=true , address=45.91.116.12:4225 , piece_id=SYPMZDE4EV26DOB6VETJKF6W3QHPBOWICPOZJTY3MV3RZOTXP3FQ, seg_pos={0 0}, upload=516526]: Long tail cancellation due to optimal threshold Dec 30 04:42:26.326807995 (589h13m26.19789106s) [noise=true , address=77.90.31.14:4663 , piece_id=3IMLBWQJQ5ONGNVYKH2ZANY5CA32T2IRW2B4I2EN3C2X3J7METZA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:26.326818175 (589h13m26.19790122s) [noise=true , address=77.90.31.14:4663 , piece_id=3IMLBWQJQ5ONGNVYKH2ZANY5CA32T2IRW2B4I2EN3C2X3J7METZA, seg_pos={0 0}, upload=516526]: Long tail cancellation due to optimal threshold Dec 30 04:42:26.326844344 (589h13m26.197927389s) [seg_pos={0 0}, upload=516526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:42:26.327014358 (589h13m26.198097403s) [upload=516526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:42:26.327023076 (589h13m26.198106111s) [upload=516526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:42:26.618365065 (589h13m26.4894481s) [upload=516526]: Done uploading segments... Dec 30 04:42:26.618391044 (589h13m26.489474089s) [upload=516526]: Upload finished. err: <nil> Duration: 16.106917484s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......XX..X.....X..............X................X..........] duration: 3.444642298s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:43:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444674349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..................................................X] duration: 1m0.003755825s trace: https://jaeger.dev.storj.io/trace/72f266853017e859 FAIL (2025-12-30 04:42:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918105148 (589h12m58.789188193s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918816583 (589h12m58.789899628s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.918855968 (589h12m58.789939013s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919168112 (589h12m58.790251147s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919392008 (589h12m58.790475073s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919460218 (589h12m58.790543253s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.919892150 (589h12m58.790975195s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920417591 (589h12m58.791500636s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920429804 (589h12m58.79151285s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.920940947 (589h12m58.792023992s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921004228 (589h12m58.792087263s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921398579 (589h12m58.792481624s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.921783332 (589h12m58.792866367s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:41:58.922187882 (589h12m58.793270917s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Uploading piece... Dec 30 04:42:00.276402715 (589h13m0.14748576s) [upload=516516]: Next returned nil segment Dec 30 04:42:00.276416642 (589h13m0.147499677s) [upload=516516]: Waiting for error group managing segments... Dec 30 04:42:00.622019383 (589h13m0.493102418s) [noise=true , address=64.49.13.170:4721 , piece_id=CY7252ZXWLIZ7WJX34AICCCNGHDCHHZZA4FNMVBKOHMWWUOZX4IQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.834416780 (589h13m0.705499826s) [noise=true , address=145.79.227.43:4186 , piece_id=NXZNFWDJJE7YCYEIH3R4L6EPIQFEKUEWO4QC2FJ4323PNTBJOVYQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.882545044 (589h13m0.75362809s) [noise=true , address=145.79.227.33:4878 , piece_id=5NKBVEG32CN5UCVEQAR6SZL5X7ZJAHFWF2YNEXSAYGPV2S5JF6RQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.933380432 (589h13m0.804463478s) [noise=true , address=145.79.227.29:4945 , piece_id=72IYCTTNHB323SNQLDTFW36MY6ALS6DB4F6K2DBUTS2RIG27URTA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.947248409 (589h13m0.818331454s) [noise=true , address=64.49.13.137:4501 , piece_id=ZN6SSGPJ325LRKTCOE4EGAJUOOHEQRYDR6DP6SO7YG564FOI7QVA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.955957202 (589h13m0.827040237s) [noise=true , address=77.90.31.12:4601 , piece_id=YQV52APO45C3PEPKVR4X4TYNPDNKN4PCK5BXGAJP2WSZGYNHKPPQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:00.993401712 (589h13m0.864484777s) [noise=true , address=145.79.227.49:4362 , piece_id=P6OVQX2RAO36LGL2YHXWCIJ3YH263IYSK3OP7Q7PBIVFXDJPJUEQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.070384730 (589h13m0.941467775s) [noise=true , address=77.90.31.10:4551 , piece_id=JW6D53LFCVRYYX6JSYJMGAGQYJ4MS64QDEG7NWH54FW4XYSUJ45A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.490289353 (589h13m1.361372398s) [noise=true , address=193.149.164.143:4701 , piece_id=L7PD57TUW5MXE2O4HWP6HDNW75TWUEXKLSWU47E5PKT2B3GPEFBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.624017151 (589h13m1.495100196s) [noise=true , address=145.79.227.34:4927 , piece_id=SRNTTQJHB36N6FTDMB266NHRGLEKJNH3J2KZDXJNM72IH55CHRMA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.625117205 (589h13m1.49620025s) [noise=true , address=77.90.31.16:4739 , piece_id=NBCZSS7HZNOFZWAHPLASJDBCBZU3CXYFXTRX7BBIA6HSQF6XEEBA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.665407685 (589h13m1.53649075s) [noise=true , address=145.79.227.44:4214 , piece_id=HN3MFEQS2M2ANSIOIMD6DUWG2JEQC6C62C33CR747IKQDAPQTK2A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.687750499 (589h13m1.558833554s) [noise=true , address=145.79.227.30:4767 , piece_id=OD5VOCYIFPSHB23OURQCG6IRIMSRVCSNE2QGQRJDQIZWYTWDUIEA, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689061385 (589h13m1.56014443s) [noise=true , address=145.79.227.40:4067 , piece_id=RSI3KNKH6OYWVRLYH4DMUPNATSDQNXVBQ42BWWCU5OOEOFRMPX5A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: <nil> Dec 30 04:42:01.689077416 (589h13m1.560160461s) [seg_pos={0 0}, upload=516516]: Segment reached optimal threshold of 14 pieces. Dec 30 04:42:01.689310499 (589h13m1.560393555s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48980->64.49.13.169:4716: use of closed network connection Dec 30 04:42:01.689319697 (589h13m1.560402742s) [noise=true , address=64.49.13.169:4716 , piece_id=VE4RMFZ3E5TRM767JTGT4WIWLEUQTYY53AHCOOPCP3RQYT47KA7A, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689470444 (589h13m1.56055352s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43276->64.49.13.169:4709: use of closed network connection Dec 30 04:42:01.689519187 (589h13m1.560602232s) [noise=true , address=64.49.13.169:4709 , piece_id=5SITRARBBRCKOHFAL7PI6EKFNT24O5WK6Q44TQME2HCAOGJYN3OQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689634196 (589h13m1.560717252s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: read tcp 188.166.225.170:35826->45.91.116.10:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:42:01.689644616 (589h13m1.560727661s) [noise=true , address=45.91.116.10:4090 , piece_id=7ESGQ6HAO56JRVD6TWOJIJAXMZNX4PNF4KKGWH3HVG5D2XOBDQ6Q, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.689927735 (589h13m1.56101077s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:01.689936862 (589h13m1.561019897s) [noise=true , address=145.79.227.35:4503 , piece_id=UOH2M64HOVR5DXSNHDVINP6KBLLW5BYJXSDWFLDZIKE4NGJ4XZXQ, seg_pos={0 0}, upload=516516]: Long tail cancellation due to optimal threshold Dec 30 04:42:01.690061630 (589h13m1.561144675s) [seg_pos={0 0}, upload=516516]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:42:01.690237415 (589h13m1.56132046s) [upload=516516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:42:01.690246662 (589h13m1.561329707s) [upload=516516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:42:01.980465043 (589h13m1.851548088s) [upload=516516]: Done uploading segments... Dec 30 04:42:01.980496303 (589h13m1.851579348s) [upload=516516]: Upload finished. err: <nil> Duration: 1m0.0037688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....X] duration: 30.004242965s trace: https://jaeger.dev.storj.io/trace/35f72752c1d7bd4b FAIL (2025-12-30 04:42:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:42:23.754942136 (589h13m23.626025171s) [upload=516526]: Starting upload Dec 30 04:42:23.754997481 (589h13m23.626080516s) [upload=516526]: Done starting upload Dec 30 04:42:23.755191541 (589h13m23.626274576s) [upload=516526]: Uploading segments... Dec 30 04:42:23.755228381 (589h13m23.626311426s) [upload=516526]: Got next segment. Inline: false Dec 30 04:42:23.755253549 (589h13m23.626336584s) [upload=516526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:42:23.755257526 (589h13m23.626340561s) [upload=516526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:42:24.006723045 (589h13m23.87780609s) [seg_pos={0 0}, upload=516526]: Begin upload segment... Dec 30 04:42:24.006794662 (589h13m23.877877707s) [seg_pos={0 0}, upload=516526]: Done begin upload segment. Dec 30 04:42:24.006914370 (589h13m23.877997445s) [noise=true , address=145.79.227.2:4114 , piece_id=PJHNJVO23H4RMMLP53I3GQ3MPW5I54XARNRN32FNUUKY6L5WTNEA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.007678395 (589h13m23.87876143s) [noise=true , address=145.79.227.47:4267 , piece_id=CP3DOZXWAEAQMMBBXCXNBL4WAE7ETPQPHZSJHEVS4X4BFMXNTTCA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.008522763 (589h13m23.879605798s) [noise=true , address=145.79.227.37:4566 , piece_id=NFWG4IZSLYROX7WEKZUDHBULY4HLWB3AVOGITTGYIQNS4MLW4NPA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.008863952 (589h13m23.879947028s) [noise=true , address=45.91.116.8:4046 , piece_id=R473EV2AC7UCHYG5RSQMHFDF2ZIGGSBN3KV7BU6KASVEM43XUAUA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.009090754 (589h13m23.880173799s) [noise=true , address=145.79.227.42:4121 , piece_id=2GSQFKSUI4TQYTMYQTRRYSOGNCZ6NLWOM6KNFU4ARYTW3MGNPTTA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.009340290 (589h13m23.880423335s) [noise=true , address=145.79.227.51:4409 , piece_id=UJC3734NTIPXZ45C4ZFMMJ2KYVA5XEEI4O6UZUZPBGU7K3PIHZAQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.009584024 (589h13m23.880667059s) [noise=true , address=145.79.227.13:4802 , piece_id=ZBFQJPYVVIEBSTAHKBE4M4UVLG6FXMN64MEHSYOIGQFZAAMFIT3A, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.009855710 (589h13m23.880938756s) [noise=true , address=145.79.227.45:4230 , piece_id=4VVGLT4B7LHRFYX7ZK3LSJDCGGSGAPMKJKN3UKATFPCTI463UDNQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.010362376 (589h13m23.881445411s) [noise=true , address=145.79.227.17:4971 , piece_id=UZ3HJKZ445REIZMGZRFC52LPCTDOSE7UFL5JI3A36IKHE2KRP2CA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.011016832 (589h13m23.882099877s) [noise=true , address=77.90.31.14:4663 , piece_id=3IMLBWQJQ5ONGNVYKH2ZANY5CA32T2IRW2B4I2EN3C2X3J7METZA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.011464804 (589h13m23.882547849s) [noise=true , address=45.91.116.12:4225 , piece_id=SYPMZDE4EV26DOB6VETJKF6W3QHPBOWICPOZJTY3MV3RZOTXP3FQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.011894293 (589h13m23.882977348s) [noise=true , address=77.90.31.17:4770 , piece_id=GRIPENYH3JRANVO3IBOVSO7UBQGDAUVN2HGNLZXW5AKPHX53V6WA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.012204063 (589h13m23.883287108s) [noise=true , address=145.79.227.40:4049 , piece_id=7QNC6DTSVAIT3FLGJJUX3III6KOZ7DFTXK73GYNXRWE52IZ2CXPQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.012855583 (589h13m23.883938628s) [noise=true , address=145.79.227.3:4151 , piece_id=NM4MK3FBBIGBXJFWGVDJONV5UZOL4ET4COULEJEXKDR6EJ22QATA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.013297494 (589h13m23.884380539s) [noise=true , address=77.90.31.21:4880 , piece_id=OFZPIKNFMGEORXFBTE63F4QZNWVZSTOODTTOEFQPHXNDZTBKE3DQ, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:24.013759574 (589h13m23.884842619s) [noise=true , address=145.79.227.50:4374 , piece_id=P2L5LDFBQFNIHZU3PV7MDAYSAOQSECU6TTFDRLUUJLQOPZ34GJIA, seg_pos={0 0}, upload=516526]: Uploading piece... Dec 30 04:42:25.614763512 (589h13m25.485846557s) [upload=516526]: Next returned nil segment Dec 30 04:42:25.614779503 (589h13m25.485862548s) [upload=516526]: Waiting for error group managing segments... Dec 30 04:42:25.994462638 (589h13m25.865545673s) [noise=true , address=145.79.227.2:4114 , piece_id=PJHNJVO23H4RMMLP53I3GQ3MPW5I54XARNRN32FNUUKY6L5WTNEA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:25.999413976 (589h13m25.870497011s) [noise=true , address=145.79.227.50:4374 , piece_id=P2L5LDFBQFNIHZU3PV7MDAYSAOQSECU6TTFDRLUUJLQOPZ34GJIA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:25.999830249 (589h13m25.870913294s) [noise=true , address=145.79.227.45:4230 , piece_id=4VVGLT4B7LHRFYX7ZK3LSJDCGGSGAPMKJKN3UKATFPCTI463UDNQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.004947744 (589h13m25.876030789s) [noise=true , address=145.79.227.47:4267 , piece_id=CP3DOZXWAEAQMMBBXCXNBL4WAE7ETPQPHZSJHEVS4X4BFMXNTTCA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.005538118 (589h13m25.876621153s) [noise=true , address=145.79.227.51:4409 , piece_id=UJC3734NTIPXZ45C4ZFMMJ2KYVA5XEEI4O6UZUZPBGU7K3PIHZAQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.007606777 (589h13m25.878689822s) [noise=true , address=145.79.227.17:4971 , piece_id=UZ3HJKZ445REIZMGZRFC52LPCTDOSE7UFL5JI3A36IKHE2KRP2CA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.015592954 (589h13m25.886676029s) [noise=true , address=145.79.227.3:4151 , piece_id=NM4MK3FBBIGBXJFWGVDJONV5UZOL4ET4COULEJEXKDR6EJ22QATA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.020051263 (589h13m25.891138877s) [noise=true , address=145.79.227.37:4566 , piece_id=NFWG4IZSLYROX7WEKZUDHBULY4HLWB3AVOGITTGYIQNS4MLW4NPA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.021901977 (589h13m25.892985022s) [noise=true , address=145.79.227.13:4802 , piece_id=ZBFQJPYVVIEBSTAHKBE4M4UVLG6FXMN64MEHSYOIGQFZAAMFIT3A, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.038871139 (589h13m25.909954184s) [noise=true , address=145.79.227.42:4121 , piece_id=2GSQFKSUI4TQYTMYQTRRYSOGNCZ6NLWOM6KNFU4ARYTW3MGNPTTA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.185232855 (589h13m26.05631589s) [noise=true , address=77.90.31.17:4770 , piece_id=GRIPENYH3JRANVO3IBOVSO7UBQGDAUVN2HGNLZXW5AKPHX53V6WA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.215356966 (589h13m26.086440001s) [noise=true , address=45.91.116.8:4046 , piece_id=R473EV2AC7UCHYG5RSQMHFDF2ZIGGSBN3KV7BU6KASVEM43XUAUA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.326015306 (589h13m26.197098342s) [noise=true , address=77.90.31.21:4880 , piece_id=OFZPIKNFMGEORXFBTE63F4QZNWVZSTOODTTOEFQPHXNDZTBKE3DQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: <nil> Dec 30 04:42:26.326051115 (589h13m26.19713415s) [seg_pos={0 0}, upload=516526]: Segment reached optimal threshold of 13 pieces. Dec 30 04:42:26.326254251 (589h13m26.197337286s) [noise=true , address=145.79.227.40:4049 , piece_id=7QNC6DTSVAIT3FLGJJUX3III6KOZ7DFTXK73GYNXRWE52IZ2CXPQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:26.326261926 (589h13m26.197344961s) [noise=true , address=145.79.227.40:4049 , piece_id=7QNC6DTSVAIT3FLGJJUX3III6KOZ7DFTXK73GYNXRWE52IZ2CXPQ, seg_pos={0 0}, upload=516526]: Long tail cancellation due to optimal threshold Dec 30 04:42:26.326547259 (589h13m26.197630304s) [noise=true , address=45.91.116.12:4225 , piece_id=SYPMZDE4EV26DOB6VETJKF6W3QHPBOWICPOZJTY3MV3RZOTXP3FQ, seg_pos={0 0}, upload=516526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:32938->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:32938->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32938->45.91.116.12:4225: use of closed network connection Dec 30 04:42:26.326566565 (589h13m26.1976496s) [noise=true , address=45.91.116.12:4225 , piece_id=SYPMZDE4EV26DOB6VETJKF6W3QHPBOWICPOZJTY3MV3RZOTXP3FQ, seg_pos={0 0}, upload=516526]: Long tail cancellation due to optimal threshold Dec 30 04:42:26.326807995 (589h13m26.19789106s) [noise=true , address=77.90.31.14:4663 , piece_id=3IMLBWQJQ5ONGNVYKH2ZANY5CA32T2IRW2B4I2EN3C2X3J7METZA, seg_pos={0 0}, upload=516526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 30 04:42:26.326818175 (589h13m26.19790122s) [noise=true , address=77.90.31.14:4663 , piece_id=3IMLBWQJQ5ONGNVYKH2ZANY5CA32T2IRW2B4I2EN3C2X3J7METZA, seg_pos={0 0}, upload=516526]: Long tail cancellation due to optimal threshold Dec 30 04:42:26.326844344 (589h13m26.197927389s) [seg_pos={0 0}, upload=516526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:42:26.327014358 (589h13m26.198097403s) [upload=516526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:42:26.327023076 (589h13m26.198106111s) [upload=516526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:42:26.618365065 (589h13m26.4894481s) [upload=516526]: Done uploading segments... Dec 30 04:42:26.618391044 (589h13m26.489474089s) [upload=516526]: Upload finished. err: <nil> Duration: 30.004293911s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........XX..X.....X..............X................X.........] duration: 3.35307858s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:42:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.353107503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........XX..X.....X..............X................X........] duration: 4.406822911s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:42:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.406851995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 47.74448461s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:41:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 47.744502655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X....] duration: 10.706512758s trace: https://jaeger.dev.storj.io/trace/31c8d3534abce85d FAIL (2025-12-30 04:41:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:38:10.816081660 (589h9m10.687164695s) [upload=516467]: Starting upload Dec 30 04:38:10.816132537 (589h9m10.687215572s) [upload=516467]: Done starting upload Dec 30 04:38:10.816205756 (589h9m10.687288791s) [upload=516467]: Uploading segments... Dec 30 04:38:10.816400457 (589h9m10.687483502s) [upload=516467]: Got next segment. Inline: false Dec 30 04:38:10.816443248 (589h9m10.687526293s) [upload=516467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:38:10.816448338 (589h9m10.687531373s) [upload=516467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:38:11.060970335 (589h9m10.93205339s) [seg_pos={0 0}, upload=516467]: Begin upload segment... Dec 30 04:38:11.061036431 (589h9m10.932119466s) [seg_pos={0 0}, upload=516467]: Done begin upload segment. Dec 30 04:38:11.061083780 (589h9m10.932166826s) [noise=true , address=145.79.227.11:4729 , piece_id=QG6G5PIOJAEOAKKR2XWPZQX7THIBJSVMO7FMDVFGXTUUT5XZ2WUQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061098339 (589h9m10.932181374s) [noise=true , address=145.79.227.36:4518 , piece_id=WILQYQEREY7DXAH7FD7TBYVHVIJWKZDC3QYHXXZCRDUHDMGWRLQA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061687060 (589h9m10.932770095s) [noise=true , address=145.79.227.47:4266 , piece_id=P5NOB6ESB52UMY6XN7F6DJPYDRQQIEIRLJSKRTHRM5WYLN4MPMQQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061879125 (589h9m10.93296217s) [noise=true , address=145.79.227.19:4079 , piece_id=ZC77YVC62INSJVFTXM36M2LNTQLRWWU6R3ADOF2EXUR3LVHIIW4Q, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062263778 (589h9m10.933346813s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062587094 (589h9m10.933670139s) [noise=true , address=145.79.227.8:4329 , piece_id=SVSA7X3IGPHHRM5YXN3AHYC5LN4JOHHZ6FW6KYGC652K4NQN4DXQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062949684 (589h9m10.934032719s) [noise=true , address=77.90.31.17:4773 , piece_id=JZ6FMH5GHX7U5Q3YEYMM7XCYYU6PHNN5BC4TAIAM6BQVFHCDBKLQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062988798 (589h9m10.934071833s) [noise=true , address=145.79.227.14:4838 , piece_id=AW72FXUZ4ATWITHXP75BMYRXAQBDJT25QITXSUDBBVJKF5ESBNRQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063277818 (589h9m10.934360863s) [noise=true , address=145.79.227.39:4013 , piece_id=FPDUWU2IIDXJQ6QPG4F4PXB3K3LNCPU4XNZIPMHEEMR2TEXFUPWA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063469313 (589h9m10.934552359s) [noise=true , address=145.79.227.3:4150 , piece_id=2PXZL6BER3Q6RJZFL4J27GVTTODHB5JHXEYBAUET7E572NMXTGOQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063604210 (589h9m10.934687245s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063913730 (589h9m10.934996765s) [noise=true , address=45.91.116.14:4192 , piece_id=WERV5XVAIESOTIZRFX6DTZ5RL3NSMADMPRKCI3P6KH2EWGCLESYA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063959757 (589h9m10.935042802s) [noise=true , address=145.79.227.22:4510 , piece_id=MIKTPDYWO3YKBWPNDERLSCQ3G3GW5DYZUZDMK4DNZIZ6ULPACTZQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.064310986 (589h9m10.935394031s) [noise=true , address=77.90.31.19:4844 , piece_id=GCI7MH36N6JWQT2JTQVJVDQN33UGNUUFDUUZQ4D74P6KK7TYTXXA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.064404574 (589h9m10.93548762s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.065623616 (589h9m10.936706661s) [noise=true , address=145.79.227.10:4402 , piece_id=GRQXFXQQJPWBKHPPCBB6TBIFYNPVG44JBCTKEF6U4PMNVYYALXCA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:12.752609615 (589h9m12.62369265s) [upload=516467]: Next returned nil segment Dec 30 04:38:12.752622059 (589h9m12.623705094s) [upload=516467]: Waiting for error group managing segments... Dec 30 04:38:13.038809534 (589h9m12.909892669s) [noise=true , address=145.79.227.36:4518 , piece_id=WILQYQEREY7DXAH7FD7TBYVHVIJWKZDC3QYHXXZCRDUHDMGWRLQA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.042049203 (589h9m12.913132238s) [noise=true , address=145.79.227.39:4013 , piece_id=FPDUWU2IIDXJQ6QPG4F4PXB3K3LNCPU4XNZIPMHEEMR2TEXFUPWA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.044525769 (589h9m12.915608814s) [noise=true , address=145.79.227.14:4838 , piece_id=AW72FXUZ4ATWITHXP75BMYRXAQBDJT25QITXSUDBBVJKF5ESBNRQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.046049440 (589h9m12.917132485s) [noise=true , address=145.79.227.22:4510 , piece_id=MIKTPDYWO3YKBWPNDERLSCQ3G3GW5DYZUZDMK4DNZIZ6ULPACTZQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.053773517 (589h9m12.924856562s) [noise=true , address=145.79.227.10:4402 , piece_id=GRQXFXQQJPWBKHPPCBB6TBIFYNPVG44JBCTKEF6U4PMNVYYALXCA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.062186097 (589h9m12.933269142s) [noise=true , address=145.79.227.8:4329 , piece_id=SVSA7X3IGPHHRM5YXN3AHYC5LN4JOHHZ6FW6KYGC652K4NQN4DXQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.065439070 (589h9m12.936522105s) [noise=true , address=145.79.227.19:4079 , piece_id=ZC77YVC62INSJVFTXM36M2LNTQLRWWU6R3ADOF2EXUR3LVHIIW4Q, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.071041408 (589h9m12.942124443s) [noise=true , address=145.79.227.11:4729 , piece_id=QG6G5PIOJAEOAKKR2XWPZQX7THIBJSVMO7FMDVFGXTUUT5XZ2WUQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.071789973 (589h9m12.942873018s) [noise=true , address=145.79.227.3:4150 , piece_id=2PXZL6BER3Q6RJZFL4J27GVTTODHB5JHXEYBAUET7E572NMXTGOQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.073222621 (589h9m12.944305666s) [noise=true , address=145.79.227.47:4266 , piece_id=P5NOB6ESB52UMY6XN7F6DJPYDRQQIEIRLJSKRTHRM5WYLN4MPMQQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.225218536 (589h9m13.096301581s) [noise=true , address=77.90.31.19:4844 , piece_id=GCI7MH36N6JWQT2JTQVJVDQN33UGNUUFDUUZQ4D74P6KK7TYTXXA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.233079163 (589h9m13.104162208s) [noise=true , address=77.90.31.17:4773 , piece_id=JZ6FMH5GHX7U5Q3YEYMM7XCYYU6PHNN5BC4TAIAM6BQVFHCDBKLQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.447002626 (589h9m13.318085661s) [noise=true , address=45.91.116.14:4192 , piece_id=WERV5XVAIESOTIZRFX6DTZ5RL3NSMADMPRKCI3P6KH2EWGCLESYA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.447023055 (589h9m13.3181061s) [seg_pos={0 0}, upload=516467]: Segment reached optimal threshold of 13 pieces. Dec 30 04:38:13.447223567 (589h9m13.318306602s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447229658 (589h9m13.318312703s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447304470 (589h9m13.318387515s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447316554 (589h9m13.318399599s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447461269 (589h9m13.318544314s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447468443 (589h9m13.318551478s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447486247 (589h9m13.318569292s) [seg_pos={0 0}, upload=516467]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:38:13.447555318 (589h9m13.318638363s) [upload=516467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:38:13.447562382 (589h9m13.318645427s) [upload=516467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:38:13.736313238 (589h9m13.607396293s) [upload=516467]: Done uploading segments... Dec 30 04:38:13.736337043 (589h9m13.607420078s) [upload=516467]: Upload finished. err: <nil> Duration: 10.706549388s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........XX..X.....X..............X................X.......] duration: 3.559159246s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:41:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559189092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X...] duration: 26.203585817s trace: https://jaeger.dev.storj.io/trace/31c8d3534abce85d FAIL (2025-12-30 04:41:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:38:10.816081660 (589h9m10.687164695s) [upload=516467]: Starting upload Dec 30 04:38:10.816132537 (589h9m10.687215572s) [upload=516467]: Done starting upload Dec 30 04:38:10.816205756 (589h9m10.687288791s) [upload=516467]: Uploading segments... Dec 30 04:38:10.816400457 (589h9m10.687483502s) [upload=516467]: Got next segment. Inline: false Dec 30 04:38:10.816443248 (589h9m10.687526293s) [upload=516467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:38:10.816448338 (589h9m10.687531373s) [upload=516467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:38:11.060970335 (589h9m10.93205339s) [seg_pos={0 0}, upload=516467]: Begin upload segment... Dec 30 04:38:11.061036431 (589h9m10.932119466s) [seg_pos={0 0}, upload=516467]: Done begin upload segment. Dec 30 04:38:11.061083780 (589h9m10.932166826s) [noise=true , address=145.79.227.11:4729 , piece_id=QG6G5PIOJAEOAKKR2XWPZQX7THIBJSVMO7FMDVFGXTUUT5XZ2WUQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061098339 (589h9m10.932181374s) [noise=true , address=145.79.227.36:4518 , piece_id=WILQYQEREY7DXAH7FD7TBYVHVIJWKZDC3QYHXXZCRDUHDMGWRLQA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061687060 (589h9m10.932770095s) [noise=true , address=145.79.227.47:4266 , piece_id=P5NOB6ESB52UMY6XN7F6DJPYDRQQIEIRLJSKRTHRM5WYLN4MPMQQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061879125 (589h9m10.93296217s) [noise=true , address=145.79.227.19:4079 , piece_id=ZC77YVC62INSJVFTXM36M2LNTQLRWWU6R3ADOF2EXUR3LVHIIW4Q, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062263778 (589h9m10.933346813s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062587094 (589h9m10.933670139s) [noise=true , address=145.79.227.8:4329 , piece_id=SVSA7X3IGPHHRM5YXN3AHYC5LN4JOHHZ6FW6KYGC652K4NQN4DXQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062949684 (589h9m10.934032719s) [noise=true , address=77.90.31.17:4773 , piece_id=JZ6FMH5GHX7U5Q3YEYMM7XCYYU6PHNN5BC4TAIAM6BQVFHCDBKLQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062988798 (589h9m10.934071833s) [noise=true , address=145.79.227.14:4838 , piece_id=AW72FXUZ4ATWITHXP75BMYRXAQBDJT25QITXSUDBBVJKF5ESBNRQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063277818 (589h9m10.934360863s) [noise=true , address=145.79.227.39:4013 , piece_id=FPDUWU2IIDXJQ6QPG4F4PXB3K3LNCPU4XNZIPMHEEMR2TEXFUPWA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063469313 (589h9m10.934552359s) [noise=true , address=145.79.227.3:4150 , piece_id=2PXZL6BER3Q6RJZFL4J27GVTTODHB5JHXEYBAUET7E572NMXTGOQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063604210 (589h9m10.934687245s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063913730 (589h9m10.934996765s) [noise=true , address=45.91.116.14:4192 , piece_id=WERV5XVAIESOTIZRFX6DTZ5RL3NSMADMPRKCI3P6KH2EWGCLESYA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063959757 (589h9m10.935042802s) [noise=true , address=145.79.227.22:4510 , piece_id=MIKTPDYWO3YKBWPNDERLSCQ3G3GW5DYZUZDMK4DNZIZ6ULPACTZQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.064310986 (589h9m10.935394031s) [noise=true , address=77.90.31.19:4844 , piece_id=GCI7MH36N6JWQT2JTQVJVDQN33UGNUUFDUUZQ4D74P6KK7TYTXXA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.064404574 (589h9m10.93548762s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.065623616 (589h9m10.936706661s) [noise=true , address=145.79.227.10:4402 , piece_id=GRQXFXQQJPWBKHPPCBB6TBIFYNPVG44JBCTKEF6U4PMNVYYALXCA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:12.752609615 (589h9m12.62369265s) [upload=516467]: Next returned nil segment Dec 30 04:38:12.752622059 (589h9m12.623705094s) [upload=516467]: Waiting for error group managing segments... Dec 30 04:38:13.038809534 (589h9m12.909892669s) [noise=true , address=145.79.227.36:4518 , piece_id=WILQYQEREY7DXAH7FD7TBYVHVIJWKZDC3QYHXXZCRDUHDMGWRLQA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.042049203 (589h9m12.913132238s) [noise=true , address=145.79.227.39:4013 , piece_id=FPDUWU2IIDXJQ6QPG4F4PXB3K3LNCPU4XNZIPMHEEMR2TEXFUPWA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.044525769 (589h9m12.915608814s) [noise=true , address=145.79.227.14:4838 , piece_id=AW72FXUZ4ATWITHXP75BMYRXAQBDJT25QITXSUDBBVJKF5ESBNRQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.046049440 (589h9m12.917132485s) [noise=true , address=145.79.227.22:4510 , piece_id=MIKTPDYWO3YKBWPNDERLSCQ3G3GW5DYZUZDMK4DNZIZ6ULPACTZQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.053773517 (589h9m12.924856562s) [noise=true , address=145.79.227.10:4402 , piece_id=GRQXFXQQJPWBKHPPCBB6TBIFYNPVG44JBCTKEF6U4PMNVYYALXCA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.062186097 (589h9m12.933269142s) [noise=true , address=145.79.227.8:4329 , piece_id=SVSA7X3IGPHHRM5YXN3AHYC5LN4JOHHZ6FW6KYGC652K4NQN4DXQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.065439070 (589h9m12.936522105s) [noise=true , address=145.79.227.19:4079 , piece_id=ZC77YVC62INSJVFTXM36M2LNTQLRWWU6R3ADOF2EXUR3LVHIIW4Q, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.071041408 (589h9m12.942124443s) [noise=true , address=145.79.227.11:4729 , piece_id=QG6G5PIOJAEOAKKR2XWPZQX7THIBJSVMO7FMDVFGXTUUT5XZ2WUQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.071789973 (589h9m12.942873018s) [noise=true , address=145.79.227.3:4150 , piece_id=2PXZL6BER3Q6RJZFL4J27GVTTODHB5JHXEYBAUET7E572NMXTGOQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.073222621 (589h9m12.944305666s) [noise=true , address=145.79.227.47:4266 , piece_id=P5NOB6ESB52UMY6XN7F6DJPYDRQQIEIRLJSKRTHRM5WYLN4MPMQQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.225218536 (589h9m13.096301581s) [noise=true , address=77.90.31.19:4844 , piece_id=GCI7MH36N6JWQT2JTQVJVDQN33UGNUUFDUUZQ4D74P6KK7TYTXXA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.233079163 (589h9m13.104162208s) [noise=true , address=77.90.31.17:4773 , piece_id=JZ6FMH5GHX7U5Q3YEYMM7XCYYU6PHNN5BC4TAIAM6BQVFHCDBKLQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.447002626 (589h9m13.318085661s) [noise=true , address=45.91.116.14:4192 , piece_id=WERV5XVAIESOTIZRFX6DTZ5RL3NSMADMPRKCI3P6KH2EWGCLESYA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.447023055 (589h9m13.3181061s) [seg_pos={0 0}, upload=516467]: Segment reached optimal threshold of 13 pieces. Dec 30 04:38:13.447223567 (589h9m13.318306602s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447229658 (589h9m13.318312703s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447304470 (589h9m13.318387515s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447316554 (589h9m13.318399599s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447461269 (589h9m13.318544314s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447468443 (589h9m13.318551478s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447486247 (589h9m13.318569292s) [seg_pos={0 0}, upload=516467]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:38:13.447555318 (589h9m13.318638363s) [upload=516467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:38:13.447562382 (589h9m13.318645427s) [upload=516467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:38:13.736313238 (589h9m13.607396293s) [upload=516467]: Done uploading segments... Dec 30 04:38:13.736337043 (589h9m13.607420078s) [upload=516467]: Upload finished. err: <nil> Duration: 26.203605224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 46.997023485s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:41:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 46.997046948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 31.97846942s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:40:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 31.978490421s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........XX..X.....X..............X................X......] duration: 4.362382342s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:40:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.362414352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 57.623837955s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:39:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 57.623854116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X..] duration: 6.105756372s trace: https://jaeger.dev.storj.io/trace/31c8d3534abce85d FAIL (2025-12-30 04:39:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:38:10.816081660 (589h9m10.687164695s) [upload=516467]: Starting upload Dec 30 04:38:10.816132537 (589h9m10.687215572s) [upload=516467]: Done starting upload Dec 30 04:38:10.816205756 (589h9m10.687288791s) [upload=516467]: Uploading segments... Dec 30 04:38:10.816400457 (589h9m10.687483502s) [upload=516467]: Got next segment. Inline: false Dec 30 04:38:10.816443248 (589h9m10.687526293s) [upload=516467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:38:10.816448338 (589h9m10.687531373s) [upload=516467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:38:11.060970335 (589h9m10.93205339s) [seg_pos={0 0}, upload=516467]: Begin upload segment... Dec 30 04:38:11.061036431 (589h9m10.932119466s) [seg_pos={0 0}, upload=516467]: Done begin upload segment. Dec 30 04:38:11.061083780 (589h9m10.932166826s) [noise=true , address=145.79.227.11:4729 , piece_id=QG6G5PIOJAEOAKKR2XWPZQX7THIBJSVMO7FMDVFGXTUUT5XZ2WUQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061098339 (589h9m10.932181374s) [noise=true , address=145.79.227.36:4518 , piece_id=WILQYQEREY7DXAH7FD7TBYVHVIJWKZDC3QYHXXZCRDUHDMGWRLQA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061687060 (589h9m10.932770095s) [noise=true , address=145.79.227.47:4266 , piece_id=P5NOB6ESB52UMY6XN7F6DJPYDRQQIEIRLJSKRTHRM5WYLN4MPMQQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061879125 (589h9m10.93296217s) [noise=true , address=145.79.227.19:4079 , piece_id=ZC77YVC62INSJVFTXM36M2LNTQLRWWU6R3ADOF2EXUR3LVHIIW4Q, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062263778 (589h9m10.933346813s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062587094 (589h9m10.933670139s) [noise=true , address=145.79.227.8:4329 , piece_id=SVSA7X3IGPHHRM5YXN3AHYC5LN4JOHHZ6FW6KYGC652K4NQN4DXQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062949684 (589h9m10.934032719s) [noise=true , address=77.90.31.17:4773 , piece_id=JZ6FMH5GHX7U5Q3YEYMM7XCYYU6PHNN5BC4TAIAM6BQVFHCDBKLQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062988798 (589h9m10.934071833s) [noise=true , address=145.79.227.14:4838 , piece_id=AW72FXUZ4ATWITHXP75BMYRXAQBDJT25QITXSUDBBVJKF5ESBNRQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063277818 (589h9m10.934360863s) [noise=true , address=145.79.227.39:4013 , piece_id=FPDUWU2IIDXJQ6QPG4F4PXB3K3LNCPU4XNZIPMHEEMR2TEXFUPWA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063469313 (589h9m10.934552359s) [noise=true , address=145.79.227.3:4150 , piece_id=2PXZL6BER3Q6RJZFL4J27GVTTODHB5JHXEYBAUET7E572NMXTGOQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063604210 (589h9m10.934687245s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063913730 (589h9m10.934996765s) [noise=true , address=45.91.116.14:4192 , piece_id=WERV5XVAIESOTIZRFX6DTZ5RL3NSMADMPRKCI3P6KH2EWGCLESYA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063959757 (589h9m10.935042802s) [noise=true , address=145.79.227.22:4510 , piece_id=MIKTPDYWO3YKBWPNDERLSCQ3G3GW5DYZUZDMK4DNZIZ6ULPACTZQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.064310986 (589h9m10.935394031s) [noise=true , address=77.90.31.19:4844 , piece_id=GCI7MH36N6JWQT2JTQVJVDQN33UGNUUFDUUZQ4D74P6KK7TYTXXA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.064404574 (589h9m10.93548762s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.065623616 (589h9m10.936706661s) [noise=true , address=145.79.227.10:4402 , piece_id=GRQXFXQQJPWBKHPPCBB6TBIFYNPVG44JBCTKEF6U4PMNVYYALXCA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:12.752609615 (589h9m12.62369265s) [upload=516467]: Next returned nil segment Dec 30 04:38:12.752622059 (589h9m12.623705094s) [upload=516467]: Waiting for error group managing segments... Dec 30 04:38:13.038809534 (589h9m12.909892669s) [noise=true , address=145.79.227.36:4518 , piece_id=WILQYQEREY7DXAH7FD7TBYVHVIJWKZDC3QYHXXZCRDUHDMGWRLQA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.042049203 (589h9m12.913132238s) [noise=true , address=145.79.227.39:4013 , piece_id=FPDUWU2IIDXJQ6QPG4F4PXB3K3LNCPU4XNZIPMHEEMR2TEXFUPWA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.044525769 (589h9m12.915608814s) [noise=true , address=145.79.227.14:4838 , piece_id=AW72FXUZ4ATWITHXP75BMYRXAQBDJT25QITXSUDBBVJKF5ESBNRQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.046049440 (589h9m12.917132485s) [noise=true , address=145.79.227.22:4510 , piece_id=MIKTPDYWO3YKBWPNDERLSCQ3G3GW5DYZUZDMK4DNZIZ6ULPACTZQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.053773517 (589h9m12.924856562s) [noise=true , address=145.79.227.10:4402 , piece_id=GRQXFXQQJPWBKHPPCBB6TBIFYNPVG44JBCTKEF6U4PMNVYYALXCA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.062186097 (589h9m12.933269142s) [noise=true , address=145.79.227.8:4329 , piece_id=SVSA7X3IGPHHRM5YXN3AHYC5LN4JOHHZ6FW6KYGC652K4NQN4DXQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.065439070 (589h9m12.936522105s) [noise=true , address=145.79.227.19:4079 , piece_id=ZC77YVC62INSJVFTXM36M2LNTQLRWWU6R3ADOF2EXUR3LVHIIW4Q, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.071041408 (589h9m12.942124443s) [noise=true , address=145.79.227.11:4729 , piece_id=QG6G5PIOJAEOAKKR2XWPZQX7THIBJSVMO7FMDVFGXTUUT5XZ2WUQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.071789973 (589h9m12.942873018s) [noise=true , address=145.79.227.3:4150 , piece_id=2PXZL6BER3Q6RJZFL4J27GVTTODHB5JHXEYBAUET7E572NMXTGOQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.073222621 (589h9m12.944305666s) [noise=true , address=145.79.227.47:4266 , piece_id=P5NOB6ESB52UMY6XN7F6DJPYDRQQIEIRLJSKRTHRM5WYLN4MPMQQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.225218536 (589h9m13.096301581s) [noise=true , address=77.90.31.19:4844 , piece_id=GCI7MH36N6JWQT2JTQVJVDQN33UGNUUFDUUZQ4D74P6KK7TYTXXA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.233079163 (589h9m13.104162208s) [noise=true , address=77.90.31.17:4773 , piece_id=JZ6FMH5GHX7U5Q3YEYMM7XCYYU6PHNN5BC4TAIAM6BQVFHCDBKLQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.447002626 (589h9m13.318085661s) [noise=true , address=45.91.116.14:4192 , piece_id=WERV5XVAIESOTIZRFX6DTZ5RL3NSMADMPRKCI3P6KH2EWGCLESYA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.447023055 (589h9m13.3181061s) [seg_pos={0 0}, upload=516467]: Segment reached optimal threshold of 13 pieces. Dec 30 04:38:13.447223567 (589h9m13.318306602s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447229658 (589h9m13.318312703s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447304470 (589h9m13.318387515s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447316554 (589h9m13.318399599s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447461269 (589h9m13.318544314s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447468443 (589h9m13.318551478s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447486247 (589h9m13.318569292s) [seg_pos={0 0}, upload=516467]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:38:13.447555318 (589h9m13.318638363s) [upload=516467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:38:13.447562382 (589h9m13.318645427s) [upload=516467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:38:13.736313238 (589h9m13.607396293s) [upload=516467]: Done uploading segments... Dec 30 04:38:13.736337043 (589h9m13.607420078s) [upload=516467]: Upload finished. err: <nil> Duration: 6.105786368s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............XX..X.....X..............X................X.....] duration: 3.5898093s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:39:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58983595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...X.] duration: 5.908060076s trace: https://jaeger.dev.storj.io/trace/31c8d3534abce85d FAIL (2025-12-30 04:39:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:38:10.816081660 (589h9m10.687164695s) [upload=516467]: Starting upload Dec 30 04:38:10.816132537 (589h9m10.687215572s) [upload=516467]: Done starting upload Dec 30 04:38:10.816205756 (589h9m10.687288791s) [upload=516467]: Uploading segments... Dec 30 04:38:10.816400457 (589h9m10.687483502s) [upload=516467]: Got next segment. Inline: false Dec 30 04:38:10.816443248 (589h9m10.687526293s) [upload=516467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:38:10.816448338 (589h9m10.687531373s) [upload=516467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:38:11.060970335 (589h9m10.93205339s) [seg_pos={0 0}, upload=516467]: Begin upload segment... Dec 30 04:38:11.061036431 (589h9m10.932119466s) [seg_pos={0 0}, upload=516467]: Done begin upload segment. Dec 30 04:38:11.061083780 (589h9m10.932166826s) [noise=true , address=145.79.227.11:4729 , piece_id=QG6G5PIOJAEOAKKR2XWPZQX7THIBJSVMO7FMDVFGXTUUT5XZ2WUQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061098339 (589h9m10.932181374s) [noise=true , address=145.79.227.36:4518 , piece_id=WILQYQEREY7DXAH7FD7TBYVHVIJWKZDC3QYHXXZCRDUHDMGWRLQA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061687060 (589h9m10.932770095s) [noise=true , address=145.79.227.47:4266 , piece_id=P5NOB6ESB52UMY6XN7F6DJPYDRQQIEIRLJSKRTHRM5WYLN4MPMQQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.061879125 (589h9m10.93296217s) [noise=true , address=145.79.227.19:4079 , piece_id=ZC77YVC62INSJVFTXM36M2LNTQLRWWU6R3ADOF2EXUR3LVHIIW4Q, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062263778 (589h9m10.933346813s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062587094 (589h9m10.933670139s) [noise=true , address=145.79.227.8:4329 , piece_id=SVSA7X3IGPHHRM5YXN3AHYC5LN4JOHHZ6FW6KYGC652K4NQN4DXQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062949684 (589h9m10.934032719s) [noise=true , address=77.90.31.17:4773 , piece_id=JZ6FMH5GHX7U5Q3YEYMM7XCYYU6PHNN5BC4TAIAM6BQVFHCDBKLQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.062988798 (589h9m10.934071833s) [noise=true , address=145.79.227.14:4838 , piece_id=AW72FXUZ4ATWITHXP75BMYRXAQBDJT25QITXSUDBBVJKF5ESBNRQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063277818 (589h9m10.934360863s) [noise=true , address=145.79.227.39:4013 , piece_id=FPDUWU2IIDXJQ6QPG4F4PXB3K3LNCPU4XNZIPMHEEMR2TEXFUPWA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063469313 (589h9m10.934552359s) [noise=true , address=145.79.227.3:4150 , piece_id=2PXZL6BER3Q6RJZFL4J27GVTTODHB5JHXEYBAUET7E572NMXTGOQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063604210 (589h9m10.934687245s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063913730 (589h9m10.934996765s) [noise=true , address=45.91.116.14:4192 , piece_id=WERV5XVAIESOTIZRFX6DTZ5RL3NSMADMPRKCI3P6KH2EWGCLESYA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.063959757 (589h9m10.935042802s) [noise=true , address=145.79.227.22:4510 , piece_id=MIKTPDYWO3YKBWPNDERLSCQ3G3GW5DYZUZDMK4DNZIZ6ULPACTZQ, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.064310986 (589h9m10.935394031s) [noise=true , address=77.90.31.19:4844 , piece_id=GCI7MH36N6JWQT2JTQVJVDQN33UGNUUFDUUZQ4D74P6KK7TYTXXA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.064404574 (589h9m10.93548762s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:11.065623616 (589h9m10.936706661s) [noise=true , address=145.79.227.10:4402 , piece_id=GRQXFXQQJPWBKHPPCBB6TBIFYNPVG44JBCTKEF6U4PMNVYYALXCA, seg_pos={0 0}, upload=516467]: Uploading piece... Dec 30 04:38:12.752609615 (589h9m12.62369265s) [upload=516467]: Next returned nil segment Dec 30 04:38:12.752622059 (589h9m12.623705094s) [upload=516467]: Waiting for error group managing segments... Dec 30 04:38:13.038809534 (589h9m12.909892669s) [noise=true , address=145.79.227.36:4518 , piece_id=WILQYQEREY7DXAH7FD7TBYVHVIJWKZDC3QYHXXZCRDUHDMGWRLQA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.042049203 (589h9m12.913132238s) [noise=true , address=145.79.227.39:4013 , piece_id=FPDUWU2IIDXJQ6QPG4F4PXB3K3LNCPU4XNZIPMHEEMR2TEXFUPWA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.044525769 (589h9m12.915608814s) [noise=true , address=145.79.227.14:4838 , piece_id=AW72FXUZ4ATWITHXP75BMYRXAQBDJT25QITXSUDBBVJKF5ESBNRQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.046049440 (589h9m12.917132485s) [noise=true , address=145.79.227.22:4510 , piece_id=MIKTPDYWO3YKBWPNDERLSCQ3G3GW5DYZUZDMK4DNZIZ6ULPACTZQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.053773517 (589h9m12.924856562s) [noise=true , address=145.79.227.10:4402 , piece_id=GRQXFXQQJPWBKHPPCBB6TBIFYNPVG44JBCTKEF6U4PMNVYYALXCA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.062186097 (589h9m12.933269142s) [noise=true , address=145.79.227.8:4329 , piece_id=SVSA7X3IGPHHRM5YXN3AHYC5LN4JOHHZ6FW6KYGC652K4NQN4DXQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.065439070 (589h9m12.936522105s) [noise=true , address=145.79.227.19:4079 , piece_id=ZC77YVC62INSJVFTXM36M2LNTQLRWWU6R3ADOF2EXUR3LVHIIW4Q, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.071041408 (589h9m12.942124443s) [noise=true , address=145.79.227.11:4729 , piece_id=QG6G5PIOJAEOAKKR2XWPZQX7THIBJSVMO7FMDVFGXTUUT5XZ2WUQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.071789973 (589h9m12.942873018s) [noise=true , address=145.79.227.3:4150 , piece_id=2PXZL6BER3Q6RJZFL4J27GVTTODHB5JHXEYBAUET7E572NMXTGOQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.073222621 (589h9m12.944305666s) [noise=true , address=145.79.227.47:4266 , piece_id=P5NOB6ESB52UMY6XN7F6DJPYDRQQIEIRLJSKRTHRM5WYLN4MPMQQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.225218536 (589h9m13.096301581s) [noise=true , address=77.90.31.19:4844 , piece_id=GCI7MH36N6JWQT2JTQVJVDQN33UGNUUFDUUZQ4D74P6KK7TYTXXA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.233079163 (589h9m13.104162208s) [noise=true , address=77.90.31.17:4773 , piece_id=JZ6FMH5GHX7U5Q3YEYMM7XCYYU6PHNN5BC4TAIAM6BQVFHCDBKLQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.447002626 (589h9m13.318085661s) [noise=true , address=45.91.116.14:4192 , piece_id=WERV5XVAIESOTIZRFX6DTZ5RL3NSMADMPRKCI3P6KH2EWGCLESYA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: <nil> Dec 30 04:38:13.447023055 (589h9m13.3181061s) [seg_pos={0 0}, upload=516467]: Segment reached optimal threshold of 13 pieces. Dec 30 04:38:13.447223567 (589h9m13.318306602s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447229658 (589h9m13.318312703s) [noise=true , address=45.91.116.11:4117 , piece_id=RZA4KZ2I7GJHAUGW2ARZ2RGUSQEMSPMLXO255AVCVH35SUNKDPMA, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447304470 (589h9m13.318387515s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447316554 (589h9m13.318399599s) [noise=true , address=77.90.31.14:4661 , piece_id=QCIAKYDUI5VM4TUURTYI6UJ4TGKN5VOB5OI6SSU6JSTW5A2NYCMQ, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447461269 (589h9m13.318544314s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 30 04:38:13.447468443 (589h9m13.318551478s) [noise=true , address=145.79.227.16:4910 , piece_id=PUIPEYHFFC772HOP3R7MYT4V2UZZRPDYNQU7XXBJTU7SQU63Q5ZA, seg_pos={0 0}, upload=516467]: Long tail cancellation due to optimal threshold Dec 30 04:38:13.447486247 (589h9m13.318569292s) [seg_pos={0 0}, upload=516467]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:38:13.447555318 (589h9m13.318638363s) [upload=516467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:38:13.447562382 (589h9m13.318645427s) [upload=516467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:38:13.736313238 (589h9m13.607396293s) [upload=516467]: Done uploading segments... Dec 30 04:38:13.736337043 (589h9m13.607420078s) [upload=516467]: Upload finished. err: <nil> Duration: 5.908081938s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)