Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 53.626837319s trace: https://jaeger.dev.storj.io/trace/160fb8586ce740e FAIL (2025-12-29 18:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7056807s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686625393 (579h19m8.557708438s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686707208 (579h19m8.557790243s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687138750 (579h19m8.558221785s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687579850 (579h19m8.558662895s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688070484 (579h19m8.559153529s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688310341 (579h19m8.559393386s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688912307 (579h19m8.559995352s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.689298552 (579h19m8.560381587s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690000118 (579h19m8.561083163s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690456467 (579h19m8.561539502s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690504810 (579h19m8.561587855s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691343126 (579h19m8.562426181s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691887513 (579h19m8.562970548s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692415137 (579h19m8.563498182s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692920369 (579h19m8.564003414s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:10.136809120 (579h19m10.007892155s) [upload=507844]: Next returned nil segment Dec 29 18:48:10.136823628 (579h19m10.007906663s) [upload=507844]: Waiting for error group managing segments... Dec 29 18:48:10.545240004 (579h19m10.416323049s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.592377642 (579h19m10.463460677s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.723127990 (579h19m10.594211026s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.726852002 (579h19m10.597935037s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.768901721 (579h19m10.639984756s) [noise=true , address=145.79.227.8:4361 , piece_id=ILEZCDJF4WZ3CDGL7XAUCXKMSVPYSNFYWDS6PYTJFZS64ALPOTQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.825937034 (579h19m10.697020069s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.888857434 (579h19m10.759940469s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.894887636 (579h19m10.765970681s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.426567354 (579h19m11.297650399s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.430877341 (579h19m11.301960386s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.437555067 (579h19m11.308638112s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.456993721 (579h19m11.328076756s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.498723440 (579h19m11.369806485s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611671490 (579h19m11.482754535s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611699082 (579h19m11.482782118s) [seg_pos={0 0}, upload=507844]: Segment reached optimal threshold of 14 pieces. Dec 29 18:48:11.612019733 (579h19m11.483102768s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g): protocol: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection Dec 29 18:48:11.612032588 (579h19m11.483115623s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612259930 (579h19m11.483342965s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612326377 (579h19m11.483409412s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612363297 (579h19m11.483446342s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612373838 (579h19m11.483456883s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612773118 (579h19m11.483856163s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612781924 (579h19m11.483864959s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612820448 (579h19m11.483903483s) [seg_pos={0 0}, upload=507844]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:48:11.612923183 (579h19m11.484006218s) [upload=507844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:48:11.612929416 (579h19m11.484012451s) [upload=507844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:48:11.898435222 (579h19m11.769518277s) [upload=507844]: Done uploading segments... Dec 29 18:48:11.898461312 (579h19m11.769544357s) [upload=507844]: Upload finished. err: <nil> Duration: 53.626866244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX..XXX] duration: 30.002903128s trace: https://jaeger.dev.storj.io/trace/11ddcdb6af39a46 FAIL (2025-12-29 18:52:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=507894]: Uploading segments... Dec 29 18:51:58.024190331 (579h22m57.895273376s) [upload=507894]: Got next segment. Inline: false Dec 29 18:51:58.024240686 (579h22m57.895323732s) [upload=507894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:51:58.024248621 (579h22m57.895331666s) [upload=507894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:51:58.272459643 (579h22m58.143542678s) [seg_pos={0 0}, upload=507894]: Begin upload segment... Dec 29 18:51:58.272505359 (579h22m58.143588394s) [seg_pos={0 0}, upload=507894]: Done begin upload segment. Dec 29 18:51:58.272536659 (579h22m58.143619704s) [noise=true , address=145.79.227.7:4293 , piece_id=6RAKN2IIGNX2W2CHXAC7JPX5LW2S7TJ56EDHEEIOZAWCJSYNLF2Q, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.272644765 (579h22m58.14372781s) [noise=true , address=45.91.116.8:4048 , piece_id=HRKD2NLX3PRNJSB6GHGFGA6HJDDWKTATE2GQ57DJHN3BSNE5SRYA, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.273215963 (579h22m58.144299008s) [noise=true , address=45.91.116.9:4262 , piece_id=6JUEUUEVF775EH2NWO6A33L33I2CKQI63DJCNZZBZC5OXDQZA6AQ, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.273393070 (579h22m58.144476105s) [noise=true , address=45.91.116.6:4973 , piece_id=FKJRZHDKU3RWWKNOS2KV65HUKQBIDARTQFX7JBUXGZ5CT6JP4X6A, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.273578273 (579h22m58.144661318s) [noise=true , address=45.91.116.11:4117 , piece_id=5MUKA3TVF6BTC3QZFPGBS4NT6FSAUDEZT3NEXNGTPPMHQ5FIXDCA, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.273978575 (579h22m58.14506162s) [noise=true , address=77.90.31.10:4517 , piece_id=PYSTJO4H5YRLHKV33D6E6YEPUUBBIXH6P4NIAZILVWSG4BGDMU7A, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.274059219 (579h22m58.145142264s) [noise=true , address=145.79.227.11:4730 , piece_id=CH77COH2U7GAYGZYIHVO7TOQYGFVNKGPF6REC3P5V2MMRWCII7NA, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.274548590 (579h22m58.145631655s) [noise=true , address=145.79.227.51:4409 , piece_id=4JNMQDLCTH4FNBM6L2S56BPHLRVLGZILD3FSW3XHTIJK4OI32MNQ, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.274601011 (579h22m58.145684056s) [noise=true , address=145.79.227.16:4912 , piece_id=HGK6DZJVX5BZBMPUQ2JVE3VT4T5BDVFYCUMCI4ODK5DP6E345YFQ, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.274898647 (579h22m58.145981692s) [noise=true , address=77.90.31.11:4553 , piece_id=7JUNA2C7ICESKPKTEO7OQJAL2WJBO2YOKWQS7INTIJTZRBF442CQ, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.275143573 (579h22m58.146226628s) [noise=true , address=45.91.116.12:4225 , piece_id=TJHGKTRKQZZBQ64YAN4IJKQ54CTAS5SDXKGI72OVQYQ3JU7ZLHIQ, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.275178770 (579h22m58.146261835s) [noise=true , address=77.90.31.13:4629 , piece_id=N2CIH6POKBF7HIPH4UWL3GIYTGJQGLLQSQXEZ35VLOY4LQVKB4CA, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.275559646 (579h22m58.146642681s) [noise=true , address=77.90.31.12:4590 , piece_id=IY6WG2WDO4RG5MJ2HQLPXXJ32T4YPFZFEQBIMW2VKTU4YXUN2XWA, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.275700464 (579h22m58.146783509s) [noise=true , address=145.79.227.15:4873 , piece_id=PEQTUG3WWKULD772QB7BJKZM7IAJ7DPNJNK6YSJEEBTDGZWKFKWA, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.276246143 (579h22m58.147329188s) [noise=true , address=145.79.227.13:4802 , piece_id=VLQVLEPMB3NSJLTXVKZ2POQQAPHILJ2QGM7ODSNNMPR25JLW2Z3Q, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:58.277063599 (579h22m58.148146634s) [noise=true , address=77.90.31.16:4736 , piece_id=ZFADYZR6UKBCDRFBHV75O6FADZ4MVICFK4R5NICYQ45PAJRFH3UQ, seg_pos={0 0}, upload=507894]: Uploading piece... Dec 29 18:51:59.975646467 (579h22m59.846729512s) [upload=507894]: Next returned nil segment Dec 29 18:51:59.975663860 (579h22m59.846746905s) [upload=507894]: Waiting for error group managing segments... Dec 29 18:52:00.257377856 (579h23m0.128460891s) [noise=true , address=145.79.227.11:4730 , piece_id=CH77COH2U7GAYGZYIHVO7TOQYGFVNKGPF6REC3P5V2MMRWCII7NA, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.260418666 (579h23m0.131501701s) [noise=true , address=145.79.227.7:4293 , piece_id=6RAKN2IIGNX2W2CHXAC7JPX5LW2S7TJ56EDHEEIOZAWCJSYNLF2Q, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.264357396 (579h23m0.135440441s) [noise=true , address=145.79.227.51:4409 , piece_id=4JNMQDLCTH4FNBM6L2S56BPHLRVLGZILD3FSW3XHTIJK4OI32MNQ, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.274386122 (579h23m0.145469168s) [noise=true , address=145.79.227.16:4912 , piece_id=HGK6DZJVX5BZBMPUQ2JVE3VT4T5BDVFYCUMCI4ODK5DP6E345YFQ, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.277352691 (579h23m0.148435726s) [noise=true , address=145.79.227.13:4802 , piece_id=VLQVLEPMB3NSJLTXVKZ2POQQAPHILJ2QGM7ODSNNMPR25JLW2Z3Q, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.281992596 (579h23m0.153075641s) [noise=true , address=145.79.227.15:4873 , piece_id=PEQTUG3WWKULD772QB7BJKZM7IAJ7DPNJNK6YSJEEBTDGZWKFKWA, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.457208364 (579h23m0.328291399s) [noise=true , address=77.90.31.10:4517 , piece_id=PYSTJO4H5YRLHKV33D6E6YEPUUBBIXH6P4NIAZILVWSG4BGDMU7A, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.573005259 (579h23m0.444088304s) [noise=true , address=77.90.31.11:4553 , piece_id=7JUNA2C7ICESKPKTEO7OQJAL2WJBO2YOKWQS7INTIJTZRBF442CQ, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.583996998 (579h23m0.455080033s) [noise=true , address=77.90.31.16:4736 , piece_id=ZFADYZR6UKBCDRFBHV75O6FADZ4MVICFK4R5NICYQ45PAJRFH3UQ, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.615221976 (579h23m0.486305011s) [noise=true , address=45.91.116.9:4262 , piece_id=6JUEUUEVF775EH2NWO6A33L33I2CKQI63DJCNZZBZC5OXDQZA6AQ, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.664735537 (579h23m0.535818572s) [noise=true , address=45.91.116.8:4048 , piece_id=HRKD2NLX3PRNJSB6GHGFGA6HJDDWKTATE2GQ57DJHN3BSNE5SRYA, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.747146240 (579h23m0.618229275s) [noise=true , address=77.90.31.12:4590 , piece_id=IY6WG2WDO4RG5MJ2HQLPXXJ32T4YPFZFEQBIMW2VKTU4YXUN2XWA, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.807502777 (579h23m0.678585822s) [noise=true , address=45.91.116.6:4973 , piece_id=FKJRZHDKU3RWWKNOS2KV65HUKQBIDARTQFX7JBUXGZ5CT6JP4X6A, seg_pos={0 0}, upload=507894]: Done uploading piece. err: <nil> Dec 29 18:52:00.807531953 (579h23m0.678614988s) [seg_pos={0 0}, upload=507894]: Segment reached optimal threshold of 13 pieces. Dec 29 18:52:00.807808188 (579h23m0.678891223s) [noise=true , address=45.91.116.11:4117 , piece_id=5MUKA3TVF6BTC3QZFPGBS4NT6FSAUDEZT3NEXNGTPPMHQ5FIXDCA, seg_pos={0 0}, upload=507894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 18:52:00.807817596 (579h23m0.678900641s) [noise=true , address=45.91.116.11:4117 , piece_id=5MUKA3TVF6BTC3QZFPGBS4NT6FSAUDEZT3NEXNGTPPMHQ5FIXDCA, seg_pos={0 0}, upload=507894]: Long tail cancellation due to optimal threshold Dec 29 18:52:00.807966700 (579h23m0.679049745s) [noise=true , address=45.91.116.12:4225 , piece_id=TJHGKTRKQZZBQ64YAN4IJKQ54CTAS5SDXKGI72OVQYQ3JU7ZLHIQ, seg_pos={0 0}, upload=507894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:47406->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:47406->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47406->45.91.116.12:4225: use of closed network connection Dec 29 18:52:00.807983292 (579h23m0.679066327s) [noise=true , address=45.91.116.12:4225 , piece_id=TJHGKTRKQZZBQ64YAN4IJKQ54CTAS5SDXKGI72OVQYQ3JU7ZLHIQ, seg_pos={0 0}, upload=507894]: Long tail cancellation due to optimal threshold Dec 29 18:52:00.808141052 (579h23m0.679224097s) [noise=true , address=77.90.31.13:4629 , piece_id=N2CIH6POKBF7HIPH4UWL3GIYTGJQGLLQSQXEZ35VLOY4LQVKB4CA, seg_pos={0 0}, upload=507894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:34798->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:34798->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34798->77.90.31.13:4629: use of closed network connection Dec 29 18:52:00.808151101 (579h23m0.679234146s) [noise=true , address=77.90.31.13:4629 , piece_id=N2CIH6POKBF7HIPH4UWL3GIYTGJQGLLQSQXEZ35VLOY4LQVKB4CA, seg_pos={0 0}, upload=507894]: Long tail cancellation due to optimal threshold Dec 29 18:52:00.808179726 (579h23m0.679262771s) [seg_pos={0 0}, upload=507894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:52:00.808310465 (579h23m0.6793935s) [upload=507894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:52:00.808318209 (579h23m0.679401254s) [upload=507894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:52:01.117616631 (579h23m0.988699666s) [upload=507894]: Done uploading segments... Dec 29 18:52:01.117647078 (579h23m0.988730113s) [upload=507894]: Upload finished. err: <nil> Duration: 30.002919228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 27.128546611s trace: https://jaeger.dev.storj.io/trace/160fb8586ce740e FAIL (2025-12-29 18:51:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7056807s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686625393 (579h19m8.557708438s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686707208 (579h19m8.557790243s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687138750 (579h19m8.558221785s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687579850 (579h19m8.558662895s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688070484 (579h19m8.559153529s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688310341 (579h19m8.559393386s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688912307 (579h19m8.559995352s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.689298552 (579h19m8.560381587s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690000118 (579h19m8.561083163s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690456467 (579h19m8.561539502s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690504810 (579h19m8.561587855s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691343126 (579h19m8.562426181s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691887513 (579h19m8.562970548s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692415137 (579h19m8.563498182s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692920369 (579h19m8.564003414s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:10.136809120 (579h19m10.007892155s) [upload=507844]: Next returned nil segment Dec 29 18:48:10.136823628 (579h19m10.007906663s) [upload=507844]: Waiting for error group managing segments... Dec 29 18:48:10.545240004 (579h19m10.416323049s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.592377642 (579h19m10.463460677s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.723127990 (579h19m10.594211026s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.726852002 (579h19m10.597935037s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.768901721 (579h19m10.639984756s) [noise=true , address=145.79.227.8:4361 , piece_id=ILEZCDJF4WZ3CDGL7XAUCXKMSVPYSNFYWDS6PYTJFZS64ALPOTQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.825937034 (579h19m10.697020069s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.888857434 (579h19m10.759940469s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.894887636 (579h19m10.765970681s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.426567354 (579h19m11.297650399s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.430877341 (579h19m11.301960386s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.437555067 (579h19m11.308638112s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.456993721 (579h19m11.328076756s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.498723440 (579h19m11.369806485s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611671490 (579h19m11.482754535s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611699082 (579h19m11.482782118s) [seg_pos={0 0}, upload=507844]: Segment reached optimal threshold of 14 pieces. Dec 29 18:48:11.612019733 (579h19m11.483102768s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g): protocol: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection Dec 29 18:48:11.612032588 (579h19m11.483115623s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612259930 (579h19m11.483342965s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612326377 (579h19m11.483409412s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612363297 (579h19m11.483446342s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612373838 (579h19m11.483456883s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612773118 (579h19m11.483856163s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612781924 (579h19m11.483864959s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612820448 (579h19m11.483903483s) [seg_pos={0 0}, upload=507844]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:48:11.612923183 (579h19m11.484006218s) [upload=507844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:48:11.612929416 (579h19m11.484012451s) [upload=507844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:48:11.898435222 (579h19m11.769518277s) [upload=507844]: Done uploading segments... Dec 29 18:48:11.898461312 (579h19m11.769544357s) [upload=507844]: Upload finished. err: <nil> Duration: 27.128562433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX..XX] duration: 30.002619328s trace: https://jaeger.dev.storj.io/trace/11dd588293942431 FAIL (2025-12-29 18:51:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: P6FNWICPPPNEELETAVWH4C7MIETCMCAF3OU675KIGOKS5VVQBFVA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: PNFWAGUS3JNWBG5HVC3H7KRIUETAQH5DYOSXFVQATTVIRFVAOCTA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: VR7XIYNP6EY7XDZXN7AA24JRQIMLP5XJOZSPVAJJZKC232SK4LZA) context deadline exceeded ======LOGS====== 29 18:50:50.606141285 (579h21m50.47722432s) [upload=507880]: Starting upload Dec 29 18:50:50.606317170 (579h21m50.477400205s) [upload=507880]: Done starting upload Dec 29 18:50:50.606383756 (579h21m50.477466802s) [upload=507880]: Uploading segments... Dec 29 18:50:50.606505307 (579h21m50.477588342s) [upload=507880]: Got next segment. Inline: false Dec 29 18:50:50.606539843 (579h21m50.477622878s) [upload=507880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:50:50.606544562 (579h21m50.477627607s) [upload=507880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:50:50.858087505 (579h21m50.72917054s) [seg_pos={0 0}, upload=507880]: Begin upload segment... Dec 29 18:50:50.858135336 (579h21m50.729218371s) [seg_pos={0 0}, upload=507880]: Done begin upload segment. Dec 29 18:50:50.858240646 (579h21m50.729323681s) [noise=true , address=145.79.227.39:4015 , piece_id=BVAM66IZFQNVSBO52BFJCGN3HEAOMV5J6APNSGTHU2FAIWQRQNZQ, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.858271756 (579h21m50.729354791s) [noise=true , address=145.79.227.15:4873 , piece_id=PFC7Q4FKRMCY3NUP43XWCNIRODIW4MPWCVUZAUE4VD3NI7G2DVIA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.858395862 (579h21m50.729478907s) [noise=true , address=45.91.116.7:4011 , piece_id=PNFWAGUS3JNWBG5HVC3H7KRIUETAQH5DYOSXFVQATTVIRFVAOCTA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.858916253 (579h21m50.729999298s) [noise=true , address=145.79.227.14:4840 , piece_id=VVSC56YYNMVRVDPEUGYRHVC5JQMUXBIKMHKXQQ4IYREK2W7TUMEQ, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.859279425 (579h21m50.73036246s) [noise=true , address=45.91.116.11:4119 , piece_id=UNKSNLGZXBQQ6VK7DJ2B7UWNF46CPKIVRCRTRSCP3CR75FUTRWWA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.859546804 (579h21m50.730629849s) [noise=true , address=77.90.31.18:4805 , piece_id=7RHBJU5ZSNF2DRNKLCCHMW2KJA4PUP6JQY7V4IMCZ6DGHHY5VDNA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.859898995 (579h21m50.73098203s) [noise=true , address=45.91.116.5:4300 , piece_id=YWHN6U7HT3BS5IOOZV6GNGKE67TE5F4SF2G25EVUF2ANGZODZAEA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.860241245 (579h21m50.731324281s) [noise=true , address=145.79.227.11:4730 , piece_id=JRB6MX52YH6ESMV2APUQQKS54B4AADN33APVLI2PFKAZOLKGRIDQ, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.860807845 (579h21m50.73189088s) [noise=true , address=77.90.31.17:4770 , piece_id=VR7XIYNP6EY7XDZXN7AA24JRQIMLP5XJOZSPVAJJZKC232SK4LZA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.861921776 (579h21m50.733004811s) [noise=true , address=45.91.116.13:4155 , piece_id=EBQQMSHGK4C7IIGOWOGA4EJYQKDICWFKALXQQ2PNDQDRUBDREHXQ, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.861976750 (579h21m50.733059815s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFA5D4F62WT5GULOFUNCXCOSSGUTZTBFAPSLOAYQSWQN7COM3N7A, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.863319097 (579h21m50.734402162s) [noise=true , address=77.90.31.10:4520 , piece_id=ELDSRWLGLX3XTJDOTEIV5OT4U3DYHB3CFGV7SV6AM4HRM6SV722Q, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.864512248 (579h21m50.735595284s) [noise=true , address=77.90.31.11:4556 , piece_id=P6FNWICPPPNEELETAVWH4C7MIETCMCAF3OU675KIGOKS5VVQBFVA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.865664853 (579h21m50.736747888s) [noise=true , address=145.79.227.42:4122 , piece_id=NB6RYL26VMHMOYF5ZKG3MND6EZPPH2MWNG5YVXXAG42DDDW24QSA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.867106829 (579h21m50.738189865s) [noise=true , address=45.91.116.14:4190 , piece_id=N7OX3RE4362OURSBJWCREEFYIDHWVR5ZSU3QEGUAR7IIAP2GUNKA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:50.868453132 (579h21m50.739536177s) [noise=true , address=145.79.227.52:4447 , piece_id=M52622OF6DSHRJPMUG6RNPG63WFGISLSZ7Z4RQVAGZ6VTPFWIGVA, seg_pos={0 0}, upload=507880]: Uploading piece... Dec 29 18:50:52.556709521 (579h21m52.427792556s) [upload=507880]: Next returned nil segment Dec 29 18:50:52.556732305 (579h21m52.42781534s) [upload=507880]: Waiting for error group managing segments... Dec 29 18:50:52.836876411 (579h21m52.707959446s) [noise=true , address=145.79.227.39:4015 , piece_id=BVAM66IZFQNVSBO52BFJCGN3HEAOMV5J6APNSGTHU2FAIWQRQNZQ, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:52.842691273 (579h21m52.713774318s) [noise=true , address=145.79.227.15:4873 , piece_id=PFC7Q4FKRMCY3NUP43XWCNIRODIW4MPWCVUZAUE4VD3NI7G2DVIA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:52.850948277 (579h21m52.722031312s) [noise=true , address=145.79.227.42:4122 , piece_id=NB6RYL26VMHMOYF5ZKG3MND6EZPPH2MWNG5YVXXAG42DDDW24QSA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:52.852383239 (579h21m52.723466274s) [noise=true , address=145.79.227.11:4730 , piece_id=JRB6MX52YH6ESMV2APUQQKS54B4AADN33APVLI2PFKAZOLKGRIDQ, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:52.870826387 (579h21m52.741909422s) [noise=true , address=145.79.227.52:4447 , piece_id=M52622OF6DSHRJPMUG6RNPG63WFGISLSZ7Z4RQVAGZ6VTPFWIGVA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:52.917003345 (579h21m52.78808639s) [noise=true , address=145.79.227.14:4840 , piece_id=VVSC56YYNMVRVDPEUGYRHVC5JQMUXBIKMHKXQQ4IYREK2W7TUMEQ, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:53.023893811 (579h21m52.894976846s) [noise=true , address=77.90.31.17:4770 , piece_id=VR7XIYNP6EY7XDZXN7AA24JRQIMLP5XJOZSPVAJJZKC232SK4LZA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:53.039579619 (579h21m52.910662654s) [noise=true , address=77.90.31.11:4556 , piece_id=P6FNWICPPPNEELETAVWH4C7MIETCMCAF3OU675KIGOKS5VVQBFVA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:53.056210215 (579h21m52.927293261s) [noise=true , address=77.90.31.18:4805 , piece_id=7RHBJU5ZSNF2DRNKLCCHMW2KJA4PUP6JQY7V4IMCZ6DGHHY5VDNA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:53.090738744 (579h21m52.961821789s) [noise=true , address=45.91.116.7:4011 , piece_id=PNFWAGUS3JNWBG5HVC3H7KRIUETAQH5DYOSXFVQATTVIRFVAOCTA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:53.119846068 (579h21m52.990929103s) [noise=true , address=77.90.31.10:4520 , piece_id=ELDSRWLGLX3XTJDOTEIV5OT4U3DYHB3CFGV7SV6AM4HRM6SV722Q, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:53.155266374 (579h21m53.026349419s) [noise=true , address=45.91.116.11:4119 , piece_id=UNKSNLGZXBQQ6VK7DJ2B7UWNF46CPKIVRCRTRSCP3CR75FUTRWWA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:53.213702334 (579h21m53.084785379s) [noise=true , address=45.91.116.5:4300 , piece_id=YWHN6U7HT3BS5IOOZV6GNGKE67TE5F4SF2G25EVUF2ANGZODZAEA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: <nil> Dec 29 18:50:53.213726962 (579h21m53.084810007s) [seg_pos={0 0}, upload=507880]: Segment reached optimal threshold of 13 pieces. Dec 29 18:50:53.213942221 (579h21m53.085025266s) [noise=true , address=45.91.116.14:4190 , piece_id=N7OX3RE4362OURSBJWCREEFYIDHWVR5ZSU3QEGUAR7IIAP2GUNKA, seg_pos={0 0}, upload=507880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 18:50:53.213949064 (579h21m53.085032099s) [noise=true , address=45.91.116.14:4190 , piece_id=N7OX3RE4362OURSBJWCREEFYIDHWVR5ZSU3QEGUAR7IIAP2GUNKA, seg_pos={0 0}, upload=507880]: Long tail cancellation due to optimal threshold Dec 29 18:50:53.214586778 (579h21m53.085669813s) [noise=true , address=45.91.116.13:4155 , piece_id=EBQQMSHGK4C7IIGOWOGA4EJYQKDICWFKALXQQ2PNDQDRUBDREHXQ, seg_pos={0 0}, upload=507880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 18:50:53.214604703 (579h21m53.085687768s) [noise=true , address=45.91.116.13:4155 , piece_id=EBQQMSHGK4C7IIGOWOGA4EJYQKDICWFKALXQQ2PNDQDRUBDREHXQ, seg_pos={0 0}, upload=507880]: Long tail cancellation due to optimal threshold Dec 29 18:50:53.214868454 (579h21m53.085951489s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFA5D4F62WT5GULOFUNCXCOSSGUTZTBFAPSLOAYQSWQN7COM3N7A, seg_pos={0 0}, upload=507880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33814->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33814->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33814->45.91.116.10:4083: use of closed network connection Dec 29 18:50:53.214878564 (579h21m53.085961619s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFA5D4F62WT5GULOFUNCXCOSSGUTZTBFAPSLOAYQSWQN7COM3N7A, seg_pos={0 0}, upload=507880]: Long tail cancellation due to optimal threshold Dec 29 18:50:53.214906888 (579h21m53.085989934s) [seg_pos={0 0}, upload=507880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:50:53.215078074 (579h21m53.086161109s) [upload=507880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:50:53.215086671 (579h21m53.086169706s) [upload=507880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:50:53.522340268 (579h21m53.393423313s) [upload=507880]: Done uploading segments... Dec 29 18:50:53.522361648 (579h21m53.393444693s) [upload=507880]: Upload finished. err: <nil> Duration: 30.002643143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 29.69730311s trace: https://jaeger.dev.storj.io/trace/160fb8586ce740e FAIL (2025-12-29 18:51:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7056807s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686625393 (579h19m8.557708438s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686707208 (579h19m8.557790243s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687138750 (579h19m8.558221785s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687579850 (579h19m8.558662895s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688070484 (579h19m8.559153529s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688310341 (579h19m8.559393386s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688912307 (579h19m8.559995352s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.689298552 (579h19m8.560381587s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690000118 (579h19m8.561083163s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690456467 (579h19m8.561539502s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690504810 (579h19m8.561587855s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691343126 (579h19m8.562426181s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691887513 (579h19m8.562970548s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692415137 (579h19m8.563498182s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692920369 (579h19m8.564003414s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:10.136809120 (579h19m10.007892155s) [upload=507844]: Next returned nil segment Dec 29 18:48:10.136823628 (579h19m10.007906663s) [upload=507844]: Waiting for error group managing segments... Dec 29 18:48:10.545240004 (579h19m10.416323049s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.592377642 (579h19m10.463460677s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.723127990 (579h19m10.594211026s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.726852002 (579h19m10.597935037s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.768901721 (579h19m10.639984756s) [noise=true , address=145.79.227.8:4361 , piece_id=ILEZCDJF4WZ3CDGL7XAUCXKMSVPYSNFYWDS6PYTJFZS64ALPOTQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.825937034 (579h19m10.697020069s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.888857434 (579h19m10.759940469s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.894887636 (579h19m10.765970681s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.426567354 (579h19m11.297650399s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.430877341 (579h19m11.301960386s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.437555067 (579h19m11.308638112s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.456993721 (579h19m11.328076756s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.498723440 (579h19m11.369806485s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611671490 (579h19m11.482754535s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611699082 (579h19m11.482782118s) [seg_pos={0 0}, upload=507844]: Segment reached optimal threshold of 14 pieces. Dec 29 18:48:11.612019733 (579h19m11.483102768s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g): protocol: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection Dec 29 18:48:11.612032588 (579h19m11.483115623s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612259930 (579h19m11.483342965s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612326377 (579h19m11.483409412s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612363297 (579h19m11.483446342s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612373838 (579h19m11.483456883s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612773118 (579h19m11.483856163s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612781924 (579h19m11.483864959s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612820448 (579h19m11.483903483s) [seg_pos={0 0}, upload=507844]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:48:11.612923183 (579h19m11.484006218s) [upload=507844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:48:11.612929416 (579h19m11.484012451s) [upload=507844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:48:11.898435222 (579h19m11.769518277s) [upload=507844]: Done uploading segments... Dec 29 18:48:11.898461312 (579h19m11.769544357s) [upload=507844]: Upload finished. err: <nil> Duration: 29.697329631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........................................................X..] duration: 31.851576464s trace: https://jaeger.dev.storj.io/trace/160fb8586ce740e FAIL (2025-12-29 18:50:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7056807s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686625393 (579h19m8.557708438s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686707208 (579h19m8.557790243s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687138750 (579h19m8.558221785s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687579850 (579h19m8.558662895s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688070484 (579h19m8.559153529s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688310341 (579h19m8.559393386s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688912307 (579h19m8.559995352s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.689298552 (579h19m8.560381587s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690000118 (579h19m8.561083163s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690456467 (579h19m8.561539502s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690504810 (579h19m8.561587855s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691343126 (579h19m8.562426181s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691887513 (579h19m8.562970548s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692415137 (579h19m8.563498182s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692920369 (579h19m8.564003414s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:10.136809120 (579h19m10.007892155s) [upload=507844]: Next returned nil segment Dec 29 18:48:10.136823628 (579h19m10.007906663s) [upload=507844]: Waiting for error group managing segments... Dec 29 18:48:10.545240004 (579h19m10.416323049s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.592377642 (579h19m10.463460677s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.723127990 (579h19m10.594211026s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.726852002 (579h19m10.597935037s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.768901721 (579h19m10.639984756s) [noise=true , address=145.79.227.8:4361 , piece_id=ILEZCDJF4WZ3CDGL7XAUCXKMSVPYSNFYWDS6PYTJFZS64ALPOTQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.825937034 (579h19m10.697020069s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.888857434 (579h19m10.759940469s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.894887636 (579h19m10.765970681s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.426567354 (579h19m11.297650399s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.430877341 (579h19m11.301960386s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.437555067 (579h19m11.308638112s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.456993721 (579h19m11.328076756s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.498723440 (579h19m11.369806485s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611671490 (579h19m11.482754535s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611699082 (579h19m11.482782118s) [seg_pos={0 0}, upload=507844]: Segment reached optimal threshold of 14 pieces. Dec 29 18:48:11.612019733 (579h19m11.483102768s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g): protocol: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection Dec 29 18:48:11.612032588 (579h19m11.483115623s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612259930 (579h19m11.483342965s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612326377 (579h19m11.483409412s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612363297 (579h19m11.483446342s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612373838 (579h19m11.483456883s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612773118 (579h19m11.483856163s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612781924 (579h19m11.483864959s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612820448 (579h19m11.483903483s) [seg_pos={0 0}, upload=507844]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:48:11.612923183 (579h19m11.484006218s) [upload=507844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:48:11.612929416 (579h19m11.484012451s) [upload=507844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:48:11.898435222 (579h19m11.769518277s) [upload=507844]: Done uploading segments... Dec 29 18:48:11.898461312 (579h19m11.769544357s) [upload=507844]: Upload finished. err: <nil> Duration: 31.851590661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX..X] duration: 30.002623863s trace: https://jaeger.dev.storj.io/trace/7dd980c148e9560d FAIL (2025-12-29 18:50:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KFDL4R3ATQPZEIGR25XYJEK47D7MFOD5UPFEGU22VA75CPLGLMHA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMW62ZETFDPU6FCK3YCTY6NLYERMQL4PNM5JWRCCHVWAR5GA3W7Q) context deadline exceeded ======LOGS====== Dec 29 18:49:38.907103507 (579h20m38.778186562s) [upload=507865]: Starting upload Dec 29 18:49:38.907175534 (579h20m38.778258589s) [upload=507865]: Done starting upload Dec 29 18:49:38.907309369 (579h20m38.778392404s) [upload=507865]: Uploading segments... Dec 29 18:49:38.907419689 (579h20m38.778502724s) [upload=507865]: Got next segment. Inline: false Dec 29 18:49:38.907466467 (579h20m38.778549512s) [upload=507865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:49:38.907472378 (579h20m38.778555413s) [upload=507865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:49:39.152110276 (579h20m39.023193311s) [seg_pos={0 0}, upload=507865]: Begin upload segment... Dec 29 18:49:39.152177564 (579h20m39.023260599s) [seg_pos={0 0}, upload=507865]: Done begin upload segment. Dec 29 18:49:39.152205397 (579h20m39.023288432s) [noise=true , address=145.79.227.49:4337 , piece_id=ZOVJV7V2XLRF64N3SMIJXJHLF64OYRQYFY35QTKHPXXQS6MYOT4Q, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.152250723 (579h20m39.023333758s) [noise=true , address=45.91.116.14:4192 , piece_id=UMW62ZETFDPU6FCK3YCTY6NLYERMQL4PNM5JWRCCHVWAR5GA3W7Q, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.152704577 (579h20m39.023787622s) [noise=true , address=145.79.227.41:4085 , piece_id=63ZJ5OFPSBJFDZ6EXDSBDT3X3UCQJIUAWUTJMT6XE42DMEWDSJ7A, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.153146128 (579h20m39.024229163s) [noise=true , address=145.79.227.51:4409 , piece_id=BFE43LDFJM5EQD44R3MTGCESIX5U4XQY5C56NQ47NCSY6YGHIVIA, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.153551831 (579h20m39.024634876s) [noise=true , address=145.79.227.17:4970 , piece_id=ATLC5BIQXGN4XYGIZGOPZNVPCJHQ4LXGSVEVYNOSXC2Y46JUOHNQ, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.153952904 (579h20m39.025035939s) [noise=true , address=145.79.227.16:4912 , piece_id=5Y2TSXK23MZOJFVE57GZFA3Z6NBRKWK6BSBOURZQFKEQONNRWAUA, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.154364478 (579h20m39.025447523s) [noise=true , address=45.91.116.9:4261 , piece_id=YZBX333VPMYGCL4G25E4OLOAHVUXAFCQVM2RQQHCLUVJJTVZ7JVA, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.154686211 (579h20m39.025769246s) [noise=true , address=145.79.227.12:4767 , piece_id=B5FA6F3IJVOOFGURH66OKOPFFT4GXFMULHLADTFLFJ4OGPXZXIZQ, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.155094388 (579h20m39.026177423s) [noise=true , address=77.90.31.16:4735 , piece_id=KFDL4R3ATQPZEIGR25XYJEK47D7MFOD5UPFEGU22VA75CPLGLMHA, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.156213619 (579h20m39.027296654s) [noise=true , address=45.91.116.12:4225 , piece_id=U4KNG5SMCB3YMURIBZTW5KN542FEGCKDJOECT4Q7V4G7KU6O6DVQ, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.156409002 (579h20m39.027492037s) [noise=true , address=145.79.227.30:4761 , piece_id=RC2ZV2ACKSOJDWHGKJ7TUJO2PONBJ7COXBBTNEQ2NHCDO7SF3ZDQ, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.157566065 (579h20m39.0286491s) [noise=true , address=77.90.31.21:4880 , piece_id=IKIRXCJPJDPAILWBFOMYUBEE4LYB4PIP73KWXS4TXVORU7QOFIHA, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.157987156 (579h20m39.029070201s) [noise=true , address=145.79.227.37:4566 , piece_id=JHUJVMDUEOMH52DTSRUORBEAJIAAM3MVEW2ZOA7V6L6SE6URZQSA, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.159164268 (579h20m39.030247303s) [noise=true , address=145.79.227.14:4839 , piece_id=SE3K4AJLYV5N4I2ULQYFKSXLI2GPXDJ26CBSHTAV25QPOGX4ZN7A, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.160310160 (579h20m39.031393195s) [noise=true , address=145.79.227.5:4222 , piece_id=L6PSZKKHV2FN3HHLWDXW5S2QQDIFLT6YDSQE7DW7NSCE3LXW33FQ, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:39.161382031 (579h20m39.032465076s) [noise=true , address=145.79.227.39:4013 , piece_id=CJUFIUILO6BMYNNYQ5DAAWFZZN6R6VHPJUVDSWDSLSTHKWKD6PRA, seg_pos={0 0}, upload=507865]: Uploading piece... Dec 29 18:49:40.849367991 (579h20m40.720451026s) [upload=507865]: Next returned nil segment Dec 29 18:49:40.849391775 (579h20m40.72047485s) [upload=507865]: Waiting for error group managing segments... Dec 29 18:49:41.139667414 (579h20m41.01075047s) [noise=true , address=145.79.227.17:4970 , piece_id=ATLC5BIQXGN4XYGIZGOPZNVPCJHQ4LXGSVEVYNOSXC2Y46JUOHNQ, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.141496407 (579h20m41.012579452s) [noise=true , address=145.79.227.37:4566 , piece_id=JHUJVMDUEOMH52DTSRUORBEAJIAAM3MVEW2ZOA7V6L6SE6URZQSA, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.144208040 (579h20m41.015291085s) [noise=true , address=145.79.227.51:4409 , piece_id=BFE43LDFJM5EQD44R3MTGCESIX5U4XQY5C56NQ47NCSY6YGHIVIA, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.148907158 (579h20m41.019990203s) [noise=true , address=145.79.227.39:4013 , piece_id=CJUFIUILO6BMYNNYQ5DAAWFZZN6R6VHPJUVDSWDSLSTHKWKD6PRA, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.160665187 (579h20m41.031748222s) [noise=true , address=145.79.227.12:4767 , piece_id=B5FA6F3IJVOOFGURH66OKOPFFT4GXFMULHLADTFLFJ4OGPXZXIZQ, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.162768252 (579h20m41.033851307s) [noise=true , address=145.79.227.16:4912 , piece_id=5Y2TSXK23MZOJFVE57GZFA3Z6NBRKWK6BSBOURZQFKEQONNRWAUA, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.163188562 (579h20m41.034271597s) [noise=true , address=145.79.227.5:4222 , piece_id=L6PSZKKHV2FN3HHLWDXW5S2QQDIFLT6YDSQE7DW7NSCE3LXW33FQ, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.168446173 (579h20m41.039529219s) [noise=true , address=145.79.227.41:4085 , piece_id=63ZJ5OFPSBJFDZ6EXDSBDT3X3UCQJIUAWUTJMT6XE42DMEWDSJ7A, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.200416279 (579h20m41.071499314s) [noise=true , address=145.79.227.14:4839 , piece_id=SE3K4AJLYV5N4I2ULQYFKSXLI2GPXDJ26CBSHTAV25QPOGX4ZN7A, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.347472537 (579h20m41.218555582s) [noise=true , address=77.90.31.16:4735 , piece_id=KFDL4R3ATQPZEIGR25XYJEK47D7MFOD5UPFEGU22VA75CPLGLMHA, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.394389985 (579h20m41.26547303s) [noise=true , address=45.91.116.14:4192 , piece_id=UMW62ZETFDPU6FCK3YCTY6NLYERMQL4PNM5JWRCCHVWAR5GA3W7Q, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.402625977 (579h20m41.273709012s) [noise=true , address=45.91.116.9:4261 , piece_id=YZBX333VPMYGCL4G25E4OLOAHVUXAFCQVM2RQQHCLUVJJTVZ7JVA, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.443076591 (579h20m41.314159636s) [noise=true , address=77.90.31.21:4880 , piece_id=IKIRXCJPJDPAILWBFOMYUBEE4LYB4PIP73KWXS4TXVORU7QOFIHA, seg_pos={0 0}, upload=507865]: Done uploading piece. err: <nil> Dec 29 18:49:41.443096730 (579h20m41.314179765s) [seg_pos={0 0}, upload=507865]: Segment reached optimal threshold of 13 pieces. Dec 29 18:49:41.443404356 (579h20m41.314487401s) [noise=true , address=145.79.227.30:4761 , piece_id=RC2ZV2ACKSOJDWHGKJ7TUJO2PONBJ7COXBBTNEQ2NHCDO7SF3ZDQ, seg_pos={0 0}, upload=507865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 18:49:41.443414054 (579h20m41.314497089s) [noise=true , address=145.79.227.30:4761 , piece_id=RC2ZV2ACKSOJDWHGKJ7TUJO2PONBJ7COXBBTNEQ2NHCDO7SF3ZDQ, seg_pos={0 0}, upload=507865]: Long tail cancellation due to optimal threshold Dec 29 18:49:41.443618233 (579h20m41.314701298s) [noise=true , address=145.79.227.49:4337 , piece_id=ZOVJV7V2XLRF64N3SMIJXJHLF64OYRQYFY35QTKHPXXQS6MYOT4Q, seg_pos={0 0}, upload=507865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 18:49:41.443628061 (579h20m41.314711106s) [noise=true , address=145.79.227.49:4337 , piece_id=ZOVJV7V2XLRF64N3SMIJXJHLF64OYRQYFY35QTKHPXXQS6MYOT4Q, seg_pos={0 0}, upload=507865]: Long tail cancellation due to optimal threshold Dec 29 18:49:41.443753781 (579h20m41.314836826s) [noise=true , address=45.91.116.12:4225 , piece_id=U4KNG5SMCB3YMURIBZTW5KN542FEGCKDJOECT4Q7V4G7KU6O6DVQ, seg_pos={0 0}, upload=507865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 18:49:41.443778988 (579h20m41.314862033s) [noise=true , address=45.91.116.12:4225 , piece_id=U4KNG5SMCB3YMURIBZTW5KN542FEGCKDJOECT4Q7V4G7KU6O6DVQ, seg_pos={0 0}, upload=507865]: Long tail cancellation due to optimal threshold Dec 29 18:49:41.443828954 (579h20m41.314911999s) [seg_pos={0 0}, upload=507865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:49:41.443985181 (579h20m41.315068226s) [upload=507865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:49:41.443991673 (579h20m41.315074718s) [upload=507865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:49:41.737315286 (579h20m41.608398331s) [upload=507865]: Done uploading segments... Dec 29 18:49:41.737337098 (579h20m41.608420133s) [upload=507865]: Upload finished. err: <nil> Duration: 30.002645234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X........................................................X.] duration: 31.61331083s trace: https://jaeger.dev.storj.io/trace/160fb8586ce740e FAIL (2025-12-29 18:49:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7056807s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686625393 (579h19m8.557708438s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686707208 (579h19m8.557790243s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687138750 (579h19m8.558221785s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687579850 (579h19m8.558662895s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688070484 (579h19m8.559153529s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688310341 (579h19m8.559393386s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688912307 (579h19m8.559995352s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.689298552 (579h19m8.560381587s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690000118 (579h19m8.561083163s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690456467 (579h19m8.561539502s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690504810 (579h19m8.561587855s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691343126 (579h19m8.562426181s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691887513 (579h19m8.562970548s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692415137 (579h19m8.563498182s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692920369 (579h19m8.564003414s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:10.136809120 (579h19m10.007892155s) [upload=507844]: Next returned nil segment Dec 29 18:48:10.136823628 (579h19m10.007906663s) [upload=507844]: Waiting for error group managing segments... Dec 29 18:48:10.545240004 (579h19m10.416323049s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.592377642 (579h19m10.463460677s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.723127990 (579h19m10.594211026s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.726852002 (579h19m10.597935037s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.768901721 (579h19m10.639984756s) [noise=true , address=145.79.227.8:4361 , piece_id=ILEZCDJF4WZ3CDGL7XAUCXKMSVPYSNFYWDS6PYTJFZS64ALPOTQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.825937034 (579h19m10.697020069s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.888857434 (579h19m10.759940469s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.894887636 (579h19m10.765970681s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.426567354 (579h19m11.297650399s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.430877341 (579h19m11.301960386s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.437555067 (579h19m11.308638112s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.456993721 (579h19m11.328076756s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.498723440 (579h19m11.369806485s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611671490 (579h19m11.482754535s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611699082 (579h19m11.482782118s) [seg_pos={0 0}, upload=507844]: Segment reached optimal threshold of 14 pieces. Dec 29 18:48:11.612019733 (579h19m11.483102768s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g): protocol: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection Dec 29 18:48:11.612032588 (579h19m11.483115623s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612259930 (579h19m11.483342965s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612326377 (579h19m11.483409412s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612363297 (579h19m11.483446342s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612373838 (579h19m11.483456883s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612773118 (579h19m11.483856163s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612781924 (579h19m11.483864959s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612820448 (579h19m11.483903483s) [seg_pos={0 0}, upload=507844]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:48:11.612923183 (579h19m11.484006218s) [upload=507844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:48:11.612929416 (579h19m11.484012451s) [upload=507844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:48:11.898435222 (579h19m11.769518277s) [upload=507844]: Done uploading segments... Dec 29 18:48:11.898461312 (579h19m11.769544357s) [upload=507844]: Upload finished. err: <nil> Duration: 31.613324347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........................................................X] duration: 1m0.007963761s trace: https://jaeger.dev.storj.io/trace/160fb8586ce740e FAIL (2025-12-29 18:49:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7056807s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686625393 (579h19m8.557708438s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.686707208 (579h19m8.557790243s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687138750 (579h19m8.558221785s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.687579850 (579h19m8.558662895s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688070484 (579h19m8.559153529s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688310341 (579h19m8.559393386s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.688912307 (579h19m8.559995352s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.689298552 (579h19m8.560381587s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690000118 (579h19m8.561083163s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690456467 (579h19m8.561539502s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.690504810 (579h19m8.561587855s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691343126 (579h19m8.562426181s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.691887513 (579h19m8.562970548s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692415137 (579h19m8.563498182s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:08.692920369 (579h19m8.564003414s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Uploading piece... Dec 29 18:48:10.136809120 (579h19m10.007892155s) [upload=507844]: Next returned nil segment Dec 29 18:48:10.136823628 (579h19m10.007906663s) [upload=507844]: Waiting for error group managing segments... Dec 29 18:48:10.545240004 (579h19m10.416323049s) [noise=true , address=145.79.227.29:4966 , piece_id=QNNYA3GGJOYYC2LVKSZZSSZ2V6SYUQRE2R6XQSBU7HZZKEV6HMRQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.592377642 (579h19m10.463460677s) [noise=true , address=145.79.227.36:4541 , piece_id=VEIYDOGCOTBQUTDKL3QIC7WAVWNQB6IWHAYRSAFCHN2VUKTROPMQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.723127990 (579h19m10.594211026s) [noise=true , address=77.90.31.14:4683 , piece_id=BHUNFCPSQ22H3HJXNG26NYPAVYM527UFMHRPS752AWJC3WMQMZZA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.726852002 (579h19m10.597935037s) [noise=true , address=5.231.73.2:4028 , piece_id=5LBNJN46W2AH7BL44EHT6LJNWNX535VOER3DUWRTO7NMOAUG4RTQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.768901721 (579h19m10.639984756s) [noise=true , address=145.79.227.8:4361 , piece_id=ILEZCDJF4WZ3CDGL7XAUCXKMSVPYSNFYWDS6PYTJFZS64ALPOTQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.825937034 (579h19m10.697020069s) [noise=true , address=77.90.31.14:4672 , piece_id=WMFCTP5KKBAXFU6FKHUYSJZV7QENOMSGX2ZWYHUBNYFLJSF6GBCA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.888857434 (579h19m10.759940469s) [noise=true , address=77.90.31.17:4803 , piece_id=KQR4HTGY27GWDWYGS7G7E76PQK2YEMMFUT7D2PHY5O7OPNPS6JWQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:10.894887636 (579h19m10.765970681s) [noise=true , address=145.79.227.16:4936 , piece_id=MWJV3QY5P74P66K57QBFJGQ5WXHUZRUPJPZFWC6KHAJ2PKYZIQ4A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.426567354 (579h19m11.297650399s) [noise=true , address=145.79.227.30:4768 , piece_id=32CPBIGD2MGOM3WHMSL5FZF3LEZLX4OSRDTO5D3NKF6EDJCZF2MA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.430877341 (579h19m11.301960386s) [noise=true , address=77.90.31.13:4640 , piece_id=RIPYVKPGZ64TRDO73F4ZZM2BAYIQDLSSDLPTW6VFFAJPGOABB4BA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.437555067 (579h19m11.308638112s) [noise=true , address=145.79.227.44:4198 , piece_id=7YJEU6CTTPL2LR3JLUT7BPXTBAWXGK25QVBDCD3YQFBRO332QHZQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.456993721 (579h19m11.328076756s) [noise=true , address=145.79.227.49:4371 , piece_id=REEJT7ZHMCG4QRJ2YOBKCWLZCZKA2FQF6GED2N23R6BPLMFD5I7A, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.498723440 (579h19m11.369806485s) [noise=true , address=145.79.227.43:4163 , piece_id=4PNRW3UIUVS2LHTJT7V5VYIN3U242P6UFBF46RXMBTGJPMLG7UXA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611671490 (579h19m11.482754535s) [noise=true , address=145.79.227.33:4873 , piece_id=VPY3Z6YNGBAK2VK5RTSNLWYD66FR7QIZLLHVTDGB5BIWJ4NFCQHA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: <nil> Dec 29 18:48:11.611699082 (579h19m11.482782118s) [seg_pos={0 0}, upload=507844]: Segment reached optimal threshold of 14 pieces. Dec 29 18:48:11.612019733 (579h19m11.483102768s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g): protocol: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40920->145.79.227.4:4204: use of closed network connection Dec 29 18:48:11.612032588 (579h19m11.483115623s) [noise=true , address=145.79.227.4:4204 , piece_id=PQO767GZHOS4PGUNXA3RC3SBREOWPKEEUEWAZQ5PP52WK2O6K4MQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612259930 (579h19m11.483342965s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612326377 (579h19m11.483409412s) [noise=true , address=77.90.31.16:4750 , piece_id=SL7L4IOSJQWRXZ7KHZODOK3PH2LZBMSBOW4ZBWFCA5ZSC6ZDLSEQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612363297 (579h19m11.483446342s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612373838 (579h19m11.483456883s) [noise=true , address=145.79.227.33:4886 , piece_id=XAGQN7CCTBJGKV4RJLRZWDXQDAWZPNQTHNDIWTBRXAWG4LJ2RPPQ, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612773118 (579h19m11.483856163s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo): protocol: expected piece hash; context canceled; EOF Dec 29 18:48:11.612781924 (579h19m11.483864959s) [noise=true , address=77.90.31.15:4728 , piece_id=QBQ526OI4EH45T2JKIS4I43ENLYJTVXGZPPQMI2AT552UTKGAZQA, seg_pos={0 0}, upload=507844]: Long tail cancellation due to optimal threshold Dec 29 18:48:11.612820448 (579h19m11.483903483s) [seg_pos={0 0}, upload=507844]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:48:11.612923183 (579h19m11.484006218s) [upload=507844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:48:11.612929416 (579h19m11.484012451s) [upload=507844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:48:11.898435222 (579h19m11.769518277s) [upload=507844]: Done uploading segments... Dec 29 18:48:11.898461312 (579h19m11.769544357s) [upload=507844]: Upload finished. err: <nil> Duration: 1m0.00798956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX..] duration: 5.686153433s trace: https://jaeger.dev.storj.io/trace/8865108ac6c41cd FAIL (2025-12-29 18:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q) context deadline exceeded ======LOGS====== Dec 29 18:46:42.441205837 (579h17m42.312288872s) [upload=507821]: Starting upload Dec 29 18:46:42.441261373 (579h17m42.312344408s) [upload=507821]: Done starting upload Dec 29 18:46:42.441411128 (579h17m42.312494163s) [upload=507821]: Uploading segments... Dec 29 18:46:42.441497002 (579h17m42.312580037s) [upload=507821]: Got next segment. Inline: false Dec 29 18:46:42.441529704 (579h17m42.312612739s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:46:42.441534223 (579h17m42.312617268s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:46:42.683954376 (579h17m42.555037442s) [seg_pos={0 0}, upload=507821]: Begin upload segment... Dec 29 18:46:42.684023749 (579h17m42.555106794s) [seg_pos={0 0}, upload=507821]: Done begin upload segment. Dec 29 18:46:42.684075667 (579h17m42.555158712s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684213550 (579h17m42.555296625s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684646804 (579h17m42.555729839s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684928071 (579h17m42.556011106s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685202202 (579h17m42.556285247s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685302613 (579h17m42.556385678s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685468539 (579h17m42.556551584s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685718826 (579h17m42.556801861s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686019589 (579h17m42.557102644s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686040379 (579h17m42.557123414s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686388171 (579h17m42.557471206s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686823189 (579h17m42.557906244s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687546166 (579h17m42.558629221s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687785712 (579h17m42.558868747s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688122132 (579h17m42.559205167s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688422364 (579h17m42.559505399s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:44.374325810 (579h17m44.245408845s) [upload=507821]: Next returned nil segment Dec 29 18:46:44.374344285 (579h17m44.24542733s) [upload=507821]: Waiting for error group managing segments... Dec 29 18:46:44.663336120 (579h17m44.534419155s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.663972081 (579h17m44.535055116s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.669270359 (579h17m44.540353424s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.680008927 (579h17m44.551091972s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.683014760 (579h17m44.554097795s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.684838594 (579h17m44.555921629s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.686348458 (579h17m44.557431503s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.689968521 (579h17m44.561051557s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.694047639 (579h17m44.565130674s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.857023028 (579h17m44.728106063s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.858769754 (579h17m44.729852789s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.917550793 (579h17m44.788633828s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921883543 (579h17m44.792966578s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921900045 (579h17m44.79298308s) [seg_pos={0 0}, upload=507821]: Segment reached optimal threshold of 13 pieces. Dec 29 18:46:44.922077011 (579h17m44.793160046s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922152325 (579h17m44.79323536s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922388925 (579h17m44.79347196s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922397311 (579h17m44.793480356s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922461954 (579h17m44.793544989s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922478305 (579h17m44.79356134s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922528621 (579h17m44.793611656s) [seg_pos={0 0}, upload=507821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:46:44.922638531 (579h17m44.793721566s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:46:44.922644632 (579h17m44.793727667s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:46:45.214309527 (579h17m45.085392572s) [upload=507821]: Done uploading segments... Dec 29 18:46:45.214339383 (579h17m45.085422428s) [upload=507821]: Upload finished. err: <nil> Duration: 5.68617858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX.] duration: 5.674427785s trace: https://jaeger.dev.storj.io/trace/8865108ac6c41cd FAIL (2025-12-29 18:48:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q) context deadline exceeded ======LOGS====== Dec 29 18:46:42.441205837 (579h17m42.312288872s) [upload=507821]: Starting upload Dec 29 18:46:42.441261373 (579h17m42.312344408s) [upload=507821]: Done starting upload Dec 29 18:46:42.441411128 (579h17m42.312494163s) [upload=507821]: Uploading segments... Dec 29 18:46:42.441497002 (579h17m42.312580037s) [upload=507821]: Got next segment. Inline: false Dec 29 18:46:42.441529704 (579h17m42.312612739s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:46:42.441534223 (579h17m42.312617268s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:46:42.683954376 (579h17m42.555037442s) [seg_pos={0 0}, upload=507821]: Begin upload segment... Dec 29 18:46:42.684023749 (579h17m42.555106794s) [seg_pos={0 0}, upload=507821]: Done begin upload segment. Dec 29 18:46:42.684075667 (579h17m42.555158712s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684213550 (579h17m42.555296625s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684646804 (579h17m42.555729839s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684928071 (579h17m42.556011106s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685202202 (579h17m42.556285247s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685302613 (579h17m42.556385678s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685468539 (579h17m42.556551584s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685718826 (579h17m42.556801861s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686019589 (579h17m42.557102644s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686040379 (579h17m42.557123414s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686388171 (579h17m42.557471206s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686823189 (579h17m42.557906244s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687546166 (579h17m42.558629221s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687785712 (579h17m42.558868747s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688122132 (579h17m42.559205167s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688422364 (579h17m42.559505399s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:44.374325810 (579h17m44.245408845s) [upload=507821]: Next returned nil segment Dec 29 18:46:44.374344285 (579h17m44.24542733s) [upload=507821]: Waiting for error group managing segments... Dec 29 18:46:44.663336120 (579h17m44.534419155s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.663972081 (579h17m44.535055116s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.669270359 (579h17m44.540353424s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.680008927 (579h17m44.551091972s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.683014760 (579h17m44.554097795s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.684838594 (579h17m44.555921629s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.686348458 (579h17m44.557431503s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.689968521 (579h17m44.561051557s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.694047639 (579h17m44.565130674s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.857023028 (579h17m44.728106063s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.858769754 (579h17m44.729852789s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.917550793 (579h17m44.788633828s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921883543 (579h17m44.792966578s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921900045 (579h17m44.79298308s) [seg_pos={0 0}, upload=507821]: Segment reached optimal threshold of 13 pieces. Dec 29 18:46:44.922077011 (579h17m44.793160046s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922152325 (579h17m44.79323536s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922388925 (579h17m44.79347196s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922397311 (579h17m44.793480356s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922461954 (579h17m44.793544989s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922478305 (579h17m44.79356134s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922528621 (579h17m44.793611656s) [seg_pos={0 0}, upload=507821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:46:44.922638531 (579h17m44.793721566s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:46:44.922644632 (579h17m44.793727667s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:46:45.214309527 (579h17m45.085392572s) [upload=507821]: Done uploading segments... Dec 29 18:46:45.214339383 (579h17m45.085422428s) [upload=507821]: Upload finished. err: <nil> Duration: 5.674456852s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 23.307545436s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:47:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 23.307584201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX] duration: 30.002468845s trace: https://jaeger.dev.storj.io/trace/8865108ac6c41cd FAIL (2025-12-29 18:47:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q) context deadline exceeded ======LOGS====== Dec 29 18:46:42.441205837 (579h17m42.312288872s) [upload=507821]: Starting upload Dec 29 18:46:42.441261373 (579h17m42.312344408s) [upload=507821]: Done starting upload Dec 29 18:46:42.441411128 (579h17m42.312494163s) [upload=507821]: Uploading segments... Dec 29 18:46:42.441497002 (579h17m42.312580037s) [upload=507821]: Got next segment. Inline: false Dec 29 18:46:42.441529704 (579h17m42.312612739s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:46:42.441534223 (579h17m42.312617268s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:46:42.683954376 (579h17m42.555037442s) [seg_pos={0 0}, upload=507821]: Begin upload segment... Dec 29 18:46:42.684023749 (579h17m42.555106794s) [seg_pos={0 0}, upload=507821]: Done begin upload segment. Dec 29 18:46:42.684075667 (579h17m42.555158712s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684213550 (579h17m42.555296625s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684646804 (579h17m42.555729839s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684928071 (579h17m42.556011106s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685202202 (579h17m42.556285247s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685302613 (579h17m42.556385678s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685468539 (579h17m42.556551584s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685718826 (579h17m42.556801861s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686019589 (579h17m42.557102644s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686040379 (579h17m42.557123414s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686388171 (579h17m42.557471206s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686823189 (579h17m42.557906244s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687546166 (579h17m42.558629221s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687785712 (579h17m42.558868747s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688122132 (579h17m42.559205167s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688422364 (579h17m42.559505399s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:44.374325810 (579h17m44.245408845s) [upload=507821]: Next returned nil segment Dec 29 18:46:44.374344285 (579h17m44.24542733s) [upload=507821]: Waiting for error group managing segments... Dec 29 18:46:44.663336120 (579h17m44.534419155s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.663972081 (579h17m44.535055116s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.669270359 (579h17m44.540353424s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.680008927 (579h17m44.551091972s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.683014760 (579h17m44.554097795s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.684838594 (579h17m44.555921629s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.686348458 (579h17m44.557431503s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.689968521 (579h17m44.561051557s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.694047639 (579h17m44.565130674s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.857023028 (579h17m44.728106063s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.858769754 (579h17m44.729852789s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.917550793 (579h17m44.788633828s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921883543 (579h17m44.792966578s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921900045 (579h17m44.79298308s) [seg_pos={0 0}, upload=507821]: Segment reached optimal threshold of 13 pieces. Dec 29 18:46:44.922077011 (579h17m44.793160046s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922152325 (579h17m44.79323536s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922388925 (579h17m44.79347196s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922397311 (579h17m44.793480356s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922461954 (579h17m44.793544989s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922478305 (579h17m44.79356134s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922528621 (579h17m44.793611656s) [seg_pos={0 0}, upload=507821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:46:44.922638531 (579h17m44.793721566s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:46:44.922644632 (579h17m44.793727667s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:46:45.214309527 (579h17m45.085392572s) [upload=507821]: Done uploading segments... Dec 29 18:46:45.214339383 (579h17m45.085422428s) [upload=507821]: Upload finished. err: <nil> Duration: 30.002494114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 15.473079618s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:47:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.473113373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 15.537370703s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:46:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.537397305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXX] duration: 30.004588709s trace: https://jaeger.dev.storj.io/trace/18a2ca15f8ebbad5 FAIL (2025-12-29 18:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DPYMF7AEO7E6LDMI7MM27F7PAVWUWP3H3J55KRC7VGCWHEXULPTQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: SHBEQUDUNTS2VSBU7WSBNGDVV5SCENXGC3BLRIG32W4SCN5DEWSA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: R2MWDEFNPMI63DXBICR3OINB7L3MYNPTXVNG7HOMILXOHPCJYMWQ) context deadline exceeded ======LOGS====== Dec 29 18:45:29.997405538 (579h16m29.868488583s) [upload=507801]: Starting upload Dec 29 18:45:29.997461184 (579h16m29.868544219s) [upload=507801]: Done starting upload Dec 29 18:45:29.997681694 (579h16m29.868764739s) [upload=507801]: Uploading segments... Dec 29 18:45:29.997721651 (579h16m29.868804686s) [upload=507801]: Got next segment. Inline: false Dec 29 18:45:29.997747018 (579h16m29.868830053s) [upload=507801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:45:29.997751326 (579h16m29.868834361s) [upload=507801]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:45:30.246555027 (579h16m30.117638062s) [seg_pos={0 0}, upload=507801]: Begin upload segment... Dec 29 18:45:30.246595785 (579h16m30.11767883s) [seg_pos={0 0}, upload=507801]: Done begin upload segment. Dec 29 18:45:30.246618116 (579h16m30.117701152s) [noise=true , address=77.90.31.12:4589 , piece_id=DPYMF7AEO7E6LDMI7MM27F7PAVWUWP3H3J55KRC7VGCWHEXULPTQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.246684263 (579h16m30.117767298s) [noise=true , address=145.79.227.12:4765 , piece_id=EXHDKYB3KSQMBDXSX46EBQPBZUGY5FYJ22G2YTGO2W66DISTLD4A, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.247372333 (579h16m30.118455368s) [noise=true , address=45.91.116.14:4192 , piece_id=SHBEQUDUNTS2VSBU7WSBNGDVV5SCENXGC3BLRIG32W4SCN5DEWSA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.247965813 (579h16m30.119048848s) [noise=true , address=145.79.227.5:4222 , piece_id=UC5LM3TSFWCW6JQOLYK5773UKU67CAN2AGEA4GMIPSPH6FI4VOXA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.248338423 (579h16m30.119421468s) [noise=true , address=45.91.116.12:4225 , piece_id=R2MWDEFNPMI63DXBICR3OINB7L3MYNPTXVNG7HOMILXOHPCJYMWQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.248689090 (579h16m30.119772136s) [noise=true , address=145.79.227.11:4729 , piece_id=SHSSR2UTR4Z5QOPABQB2SYBA3JLQME6XCKKVEGTDHGQHY7R4Y62A, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.249059857 (579h16m30.120142892s) [noise=true , address=45.91.116.4:4939 , piece_id=BKAMYPCAXF24VHJRC5UAKPCA4XOUFBKMVOWFCFVD2VIJLPVKCKDQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.249534961 (579h16m30.120618006s) [noise=true , address=145.79.227.49:4339 , piece_id=B3UJYKAMRVRWBDBDTU47GJBOC7VKZ377DQW363OKIJPT5Y6YUPJA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.250712433 (579h16m30.121795468s) [noise=true , address=45.91.116.6:4973 , piece_id=OR335JA6AUPEKUM6FV5OPYPCG5AH55DPCZDDS5VMTY6BRNM6H5EQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.252240303 (579h16m30.123323338s) [noise=true , address=145.79.227.4:4186 , piece_id=VLYTHEBWPLMEAUF5LFJ4XXS4KMQDXHZK4TMAP4BETZF4DDGVFEXA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.254282380 (579h16m30.125365415s) [noise=true , address=145.79.227.45:4229 , piece_id=HJNSE7MRTSLQWGSQCYLUCBSEDFEV3K64EB4DERKUXHLJYDR3UHIQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.258952543 (579h16m30.130035588s) [noise=true , address=145.79.227.19:4079 , piece_id=4CL7DQLWTXYVV7KQIFZWOH2AGULRFNXO2YN4MSPQAQ5P42GZH4DQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.260767809 (579h16m30.131850855s) [noise=true , address=45.91.116.9:4261 , piece_id=E323JETYPKF377M6BY7ONX2F63Z7X5C3QWPXN2UPH4OXPAN4FNBA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.264585690 (579h16m30.135668725s) [noise=true , address=145.79.227.33:4871 , piece_id=ETYZPJ3MGSVKPKJHNTCPGZDO32RWCH7HTOKSRK3QIHUP6NLSH6LA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.265216781 (579h16m30.136299826s) [noise=true , address=45.91.116.7:4009 , piece_id=VLNL4NS4LXCQ7AQNP5GPBWSRKRFS7KRJ6U5SQ62KJKTPYFS2GZJQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.268393119 (579h16m30.139476164s) [noise=true , address=45.91.116.5:4299 , piece_id=QZ6IZFTZTUJDJIT6W7G3JL7VAKRNLK35MLI6AXL5YQATHK7P5VYQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:31.851451682 (579h16m31.722534727s) [upload=507801]: Next returned nil segment Dec 29 18:45:31.851468034 (579h16m31.722551079s) [upload=507801]: Waiting for error group managing segments... Dec 29 18:45:32.234275227 (579h16m32.105358262s) [noise=true , address=145.79.227.12:4765 , piece_id=EXHDKYB3KSQMBDXSX46EBQPBZUGY5FYJ22G2YTGO2W66DISTLD4A, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.249663528 (579h16m32.120746573s) [noise=true , address=145.79.227.11:4729 , piece_id=SHSSR2UTR4Z5QOPABQB2SYBA3JLQME6XCKKVEGTDHGQHY7R4Y62A, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.261536786 (579h16m32.132619832s) [noise=true , address=145.79.227.45:4229 , piece_id=HJNSE7MRTSLQWGSQCYLUCBSEDFEV3K64EB4DERKUXHLJYDR3UHIQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.261574740 (579h16m32.132657785s) [noise=true , address=145.79.227.5:4222 , piece_id=UC5LM3TSFWCW6JQOLYK5773UKU67CAN2AGEA4GMIPSPH6FI4VOXA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.269788289 (579h16m32.140871324s) [noise=true , address=145.79.227.19:4079 , piece_id=4CL7DQLWTXYVV7KQIFZWOH2AGULRFNXO2YN4MSPQAQ5P42GZH4DQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.283356185 (579h16m32.15443923s) [noise=true , address=145.79.227.4:4186 , piece_id=VLYTHEBWPLMEAUF5LFJ4XXS4KMQDXHZK4TMAP4BETZF4DDGVFEXA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.503571559 (579h16m32.374654594s) [noise=true , address=45.91.116.7:4009 , piece_id=VLNL4NS4LXCQ7AQNP5GPBWSRKRFS7KRJ6U5SQ62KJKTPYFS2GZJQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.504458778 (579h16m32.375541813s) [noise=true , address=45.91.116.12:4225 , piece_id=R2MWDEFNPMI63DXBICR3OINB7L3MYNPTXVNG7HOMILXOHPCJYMWQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.505792748 (579h16m32.376875793s) [noise=true , address=45.91.116.14:4192 , piece_id=SHBEQUDUNTS2VSBU7WSBNGDVV5SCENXGC3BLRIG32W4SCN5DEWSA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.535311226 (579h16m32.406394271s) [noise=true , address=45.91.116.5:4299 , piece_id=QZ6IZFTZTUJDJIT6W7G3JL7VAKRNLK35MLI6AXL5YQATHK7P5VYQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.553243982 (579h16m32.424327027s) [noise=true , address=45.91.116.4:4939 , piece_id=BKAMYPCAXF24VHJRC5UAKPCA4XOUFBKMVOWFCFVD2VIJLPVKCKDQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.563846030 (579h16m32.434929075s) [noise=true , address=45.91.116.9:4261 , piece_id=E323JETYPKF377M6BY7ONX2F63Z7X5C3QWPXN2UPH4OXPAN4FNBA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.583730774 (579h16m32.454813809s) [noise=true , address=77.90.31.12:4589 , piece_id=DPYMF7AEO7E6LDMI7MM27F7PAVWUWP3H3J55KRC7VGCWHEXULPTQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.583748938 (579h16m32.454831983s) [seg_pos={0 0}, upload=507801]: Segment reached optimal threshold of 13 pieces. Dec 29 18:45:32.584000086 (579h16m32.455083131s) [noise=true , address=145.79.227.49:4339 , piece_id=B3UJYKAMRVRWBDBDTU47GJBOC7VKZ377DQW363OKIJPT5Y6YUPJA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 18:45:32.584008423 (579h16m32.455091468s) [noise=true , address=145.79.227.49:4339 , piece_id=B3UJYKAMRVRWBDBDTU47GJBOC7VKZ377DQW363OKIJPT5Y6YUPJA, seg_pos={0 0}, upload=507801]: Long tail cancellation due to optimal threshold Dec 29 18:45:32.584113132 (579h16m32.455196207s) [noise=true , address=145.79.227.33:4871 , piece_id=ETYZPJ3MGSVKPKJHNTCPGZDO32RWCH7HTOKSRK3QIHUP6NLSH6LA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 18:45:32.584135263 (579h16m32.455218319s) [noise=true , address=145.79.227.33:4871 , piece_id=ETYZPJ3MGSVKPKJHNTCPGZDO32RWCH7HTOKSRK3QIHUP6NLSH6LA, seg_pos={0 0}, upload=507801]: Long tail cancellation due to optimal threshold Dec 29 18:45:32.584316669 (579h16m32.455399714s) [noise=true , address=45.91.116.6:4973 , piece_id=OR335JA6AUPEKUM6FV5OPYPCG5AH55DPCZDDS5VMTY6BRNM6H5EQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 18:45:32.584324815 (579h16m32.45540785s) [noise=true , address=45.91.116.6:4973 , piece_id=OR335JA6AUPEKUM6FV5OPYPCG5AH55DPCZDDS5VMTY6BRNM6H5EQ, seg_pos={0 0}, upload=507801]: Long tail cancellation due to optimal threshold Dec 29 18:45:32.584352888 (579h16m32.455435923s) [seg_pos={0 0}, upload=507801]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:45:32.584430666 (579h16m32.455513712s) [upload=507801]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:45:32.584437499 (579h16m32.455520534s) [upload=507801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:45:32.878593124 (579h16m32.749676159s) [upload=507801]: Done uploading segments... Dec 29 18:45:32.878612592 (579h16m32.749695627s) [upload=507801]: Upload finished. err: <nil> Duration: 30.004606562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 27.316027635s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:45:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 27.316059725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXX] duration: 30.003691658s trace: https://jaeger.dev.storj.io/trace/286224f30dcb00dd FAIL (2025-12-29 18:44:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: SNSBK46CGFGUQGGEYUTS6PZV6J7TRC77KS4BRAW5XOCMLXXRW74A) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: OHYGUQKOJBNEZFQUKFE7VPF5J6I3CMH4DDN52GU5ISDWQWYH4UJQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: DVEQODFVDXVUG6GS5NZWIVSRWG64RPLNZWXTDCJXDXYNKWYHVXIQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VRBHPOLBYFB7LX7CHYNS56OP26N2WNWCCOI42MIFFLQUOT3L7ANA) context deadline exceeded ======LOGS====== Dec 29 18:44:29.085406105 (579h15m28.95648914s) [upload=507789]: Starting upload Dec 29 18:44:29.085459476 (579h15m28.956542511s) [upload=507789]: Done starting upload Dec 29 18:44:29.085557332 (579h15m28.956640367s) [upload=507789]: Uploading segments... Dec 29 18:44:29.085674365 (579h15m28.95675741s) [upload=507789]: Got next segment. Inline: false Dec 29 18:44:29.085735922 (579h15m28.956818967s) [upload=507789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:44:29.085742865 (579h15m28.95682591s) [upload=507789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:44:29.333736895 (579h15m29.20481996s) [seg_pos={0 0}, upload=507789]: Begin upload segment... Dec 29 18:44:29.333845321 (579h15m29.204928356s) [seg_pos={0 0}, upload=507789]: Done begin upload segment. Dec 29 18:44:29.333861521 (579h15m29.204944567s) [noise=true , address=45.91.116.10:4082 , piece_id=DVEQODFVDXVUG6GS5NZWIVSRWG64RPLNZWXTDCJXDXYNKWYHVXIQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.333887080 (579h15m29.204970145s) [noise=true , address=145.79.227.46:4602 , piece_id=VOFJUCIQNB6EE2MQDV5GCJHR7V6SPOVQY2CTK4LSJTIP5BHM4ZIQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.333955360 (579h15m29.205038406s) [noise=true , address=45.91.116.7:4009 , piece_id=NVP3OHPQTBMPUX2SFLNS3IGEAYIFLIEKF6RBSRCICNF5KR3QMBZQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.337852151 (579h15m29.208935206s) [noise=true , address=45.91.116.5:4299 , piece_id=4EGY3JZ3NSTAIAD3HUZ2OBXFK7TDBR3RGA5BYFU7LLFL7F7OTZUA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.338233527 (579h15m29.209316562s) [noise=true , address=145.79.227.21:4475 , piece_id=M6O2OWQLZVVKW4MOZIP66UMX5ACSJJYWHJVQKJVI5JOXBJVYFJXA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.338698712 (579h15m29.209781757s) [noise=true , address=145.79.227.18:4007 , piece_id=7XKEGGNN5C4CDDRNJU3TZUJGTPIDPHTVFNU3P6ABD5AJBFOLLSHA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.339875042 (579h15m29.210958088s) [noise=true , address=45.91.116.6:4975 , piece_id=VRBHPOLBYFB7LX7CHYNS56OP26N2WNWCCOI42MIFFLQUOT3L7ANA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.340809302 (579h15m29.211892337s) [noise=true , address=145.79.227.51:4409 , piece_id=AYZM3KWGBSBYQV7U7UT5JXTK5NZBKZSUPPYN2KBECZP6O4R2VMQQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.341400307 (579h15m29.212483352s) [noise=true , address=77.90.31.14:4663 , piece_id=SNSBK46CGFGUQGGEYUTS6PZV6J7TRC77KS4BRAW5XOCMLXXRW74A, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.341995530 (579h15m29.213078565s) [noise=true , address=145.79.227.19:4079 , piece_id=KQSFPXVCHA2N24EVGUKTDCO5CDGNYDYQAKU5S65JR7YCJTM5UXRA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.343709073 (579h15m29.214792118s) [noise=true , address=77.90.31.10:4518 , piece_id=OHYGUQKOJBNEZFQUKFE7VPF5J6I3CMH4DDN52GU5ISDWQWYH4UJQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.345128927 (579h15m29.216211962s) [noise=true , address=145.79.227.26:4653 , piece_id=G4B35CJC7Q6C7VRDLDOP636HPR4PSALCRPASOJXFTKKBOCZNOBLA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.346787876 (579h15m29.217870921s) [noise=true , address=45.91.116.9:4263 , piece_id=LZWVUSXBMTA5T5BP2MW4R2UUJJG5KEBPIDTHH6FEMINOFZXBDWJA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.348518511 (579h15m29.219601556s) [noise=true , address=145.79.227.50:4375 , piece_id=OMFAS6MYVSGFLW6QLSBP2L43LZJUWUI4UBMXRPAQGQNKR2PGVQIA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.350156931 (579h15m29.221239966s) [noise=true , address=45.91.116.14:4191 , piece_id=CSYC4Q6ZUCRLGFSVDNKI6UVDYTKSI4XHRQ67LC4WWBMXCHTKSYZQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.351471785 (579h15m29.22255482s) [noise=true , address=145.79.227.29:4942 , piece_id=4DRK2CQPFGLODGG7JNBJH7ZK7NEO233IEHCJBAHZOXDCDCZXY3GQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:31.034219503 (579h15m30.905302548s) [upload=507789]: Next returned nil segment Dec 29 18:44:31.034240162 (579h15m30.905323197s) [upload=507789]: Waiting for error group managing segments... Dec 29 18:44:31.314165833 (579h15m31.185248868s) [noise=true , address=145.79.227.19:4079 , piece_id=KQSFPXVCHA2N24EVGUKTDCO5CDGNYDYQAKU5S65JR7YCJTM5UXRA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.320971141 (579h15m31.192054176s) [noise=true , address=145.79.227.18:4007 , piece_id=7XKEGGNN5C4CDDRNJU3TZUJGTPIDPHTVFNU3P6ABD5AJBFOLLSHA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.321663479 (579h15m31.192746515s) [noise=true , address=145.79.227.21:4475 , piece_id=M6O2OWQLZVVKW4MOZIP66UMX5ACSJJYWHJVQKJVI5JOXBJVYFJXA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.335497972 (579h15m31.206581017s) [noise=true , address=145.79.227.46:4602 , piece_id=VOFJUCIQNB6EE2MQDV5GCJHR7V6SPOVQY2CTK4LSJTIP5BHM4ZIQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.348980736 (579h15m31.220063771s) [noise=true , address=145.79.227.26:4653 , piece_id=G4B35CJC7Q6C7VRDLDOP636HPR4PSALCRPASOJXFTKKBOCZNOBLA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.359114181 (579h15m31.230197216s) [noise=true , address=145.79.227.51:4409 , piece_id=AYZM3KWGBSBYQV7U7UT5JXTK5NZBKZSUPPYN2KBECZP6O4R2VMQQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.582331441 (579h15m31.453414486s) [noise=true , address=77.90.31.10:4518 , piece_id=OHYGUQKOJBNEZFQUKFE7VPF5J6I3CMH4DDN52GU5ISDWQWYH4UJQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.586724296 (579h15m31.457807341s) [noise=true , address=45.91.116.7:4009 , piece_id=NVP3OHPQTBMPUX2SFLNS3IGEAYIFLIEKF6RBSRCICNF5KR3QMBZQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.626614112 (579h15m31.497697167s) [noise=true , address=45.91.116.6:4975 , piece_id=VRBHPOLBYFB7LX7CHYNS56OP26N2WNWCCOI42MIFFLQUOT3L7ANA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.705343164 (579h15m31.576426209s) [noise=true , address=45.91.116.14:4191 , piece_id=CSYC4Q6ZUCRLGFSVDNKI6UVDYTKSI4XHRQ67LC4WWBMXCHTKSYZQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.729639278 (579h15m31.600722323s) [noise=true , address=45.91.116.9:4263 , piece_id=LZWVUSXBMTA5T5BP2MW4R2UUJJG5KEBPIDTHH6FEMINOFZXBDWJA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:32.462448999 (579h15m32.333532034s) [noise=true , address=45.91.116.10:4082 , piece_id=DVEQODFVDXVUG6GS5NZWIVSRWG64RPLNZWXTDCJXDXYNKWYHVXIQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:32.510668807 (579h15m32.381751842s) [noise=true , address=77.90.31.14:4663 , piece_id=SNSBK46CGFGUQGGEYUTS6PZV6J7TRC77KS4BRAW5XOCMLXXRW74A, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:32.510691250 (579h15m32.381774285s) [seg_pos={0 0}, upload=507789]: Segment reached optimal threshold of 13 pieces. Dec 29 18:44:32.510881331 (579h15m32.381964366s) [noise=true , address=145.79.227.50:4375 , piece_id=OMFAS6MYVSGFLW6QLSBP2L43LZJUWUI4UBMXRPAQGQNKR2PGVQIA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:32.510888095 (579h15m32.38197113s) [noise=true , address=145.79.227.50:4375 , piece_id=OMFAS6MYVSGFLW6QLSBP2L43LZJUWUI4UBMXRPAQGQNKR2PGVQIA, seg_pos={0 0}, upload=507789]: Long tail cancellation due to optimal threshold Dec 29 18:44:32.511071704 (579h15m32.382154739s) [noise=true , address=45.91.116.5:4299 , piece_id=4EGY3JZ3NSTAIAD3HUZ2OBXFK7TDBR3RGA5BYFU7LLFL7F7OTZUA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:32.511077565 (579h15m32.3821606s) [noise=true , address=45.91.116.5:4299 , piece_id=4EGY3JZ3NSTAIAD3HUZ2OBXFK7TDBR3RGA5BYFU7LLFL7F7OTZUA, seg_pos={0 0}, upload=507789]: Long tail cancellation due to optimal threshold Dec 29 18:44:32.511173689 (579h15m32.382256724s) [noise=true , address=145.79.227.29:4942 , piece_id=4DRK2CQPFGLODGG7JNBJH7ZK7NEO233IEHCJBAHZOXDCDCZXY3GQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:32.511190941 (579h15m32.382273976s) [noise=true , address=145.79.227.29:4942 , piece_id=4DRK2CQPFGLODGG7JNBJH7ZK7NEO233IEHCJBAHZOXDCDCZXY3GQ, seg_pos={0 0}, upload=507789]: Long tail cancellation due to optimal threshold Dec 29 18:44:32.511217963 (579h15m32.382300998s) [seg_pos={0 0}, upload=507789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:44:32.511349684 (579h15m32.382432719s) [upload=507789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:44:32.511355294 (579h15m32.382438329s) [upload=507789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:44:32.831330165 (579h15m32.70241321s) [upload=507789]: Done uploading segments... Dec 29 18:44:32.831349401 (579h15m32.702432436s) [upload=507789]: Upload finished. err: <nil> Duration: 30.003704152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXX] duration: 30.003015409s trace: https://jaeger.dev.storj.io/trace/347a0e66a4adc131 FAIL (2025-12-29 18:44:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 18:43:58.845608877 (579h14m58.716691912s) [upload=507782]: Starting upload Dec 29 18:43:58.845663341 (579h14m58.716746376s) [upload=507782]: Done starting upload Dec 29 18:43:58.845755716 (579h14m58.716838761s) [upload=507782]: Uploading segments... Dec 29 18:43:58.845904560 (579h14m58.716987595s) [upload=507782]: Got next segment. Inline: false Dec 29 18:43:58.845937944 (579h14m58.717020979s) [upload=507782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:43:58.845943474 (579h14m58.717026509s) [upload=507782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:43:59.102958720 (579h14m58.974041755s) [seg_pos={0 0}, upload=507782]: Begin upload segment... Dec 29 18:43:59.103014576 (579h14m58.974097611s) [seg_pos={0 0}, upload=507782]: Done begin upload segment. Dec 29 18:43:59.103052028 (579h14m58.974135063s) [noise=true , address=45.91.116.14:4192 , piece_id=3HWMHIGS7EFFXMSVPSQ2METAEORLEAEBS7JGX5A46DMCCCWE5OIQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.103161937 (579h14m58.974244982s) [noise=true , address=145.79.227.35:4482 , piece_id=DZ3KKNC3W7OSMFNAGAONKNILWJFH35C7622RFLP23KZDLXCRFBRA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.103864504 (579h14m58.974947549s) [noise=true , address=145.79.227.44:4194 , piece_id=5WX6VRGQNUWG3CYOH46SDEHRMAV6D5SUC6RHCRGAKNUHGBVPRFYA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.104001415 (579h14m58.97508445s) [noise=true , address=77.90.31.14:4661 , piece_id=FBMFPP3PPBCNSU2IU3I5LOYWFX4EASTPC2VFCXXPSSFIRZP3YISA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.104491899 (579h14m58.975574934s) [noise=true , address=145.79.227.17:4972 , piece_id=SQ7D3Y6FOA53Z5YRMTBCFIVQZHN6XSNUCWJHLDYIABVQ36KU4CIQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.105347008 (579h14m58.976430043s) [noise=true , address=45.91.116.9:4264 , piece_id=MLD4YWZ53STUN3QW4SSWAA5HPVFX7PFFVC45TL4QFNH4GXBPMADQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.105395700 (579h14m58.976478745s) [noise=true , address=77.90.31.11:4555 , piece_id=WLVJH6I7RY4JRSMFIPD6VFE5I37MIR6EH4MSLW5QRBNDYXM6OC5A, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.105987827 (579h14m58.977070873s) [noise=true , address=145.79.227.7:4295 , piece_id=KRYEJUVZF3TCVZERFOIEHPKKEHP36GALEICAWKHI32RS4P5PXCSA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.106422646 (579h14m58.977505681s) [noise=true , address=145.79.227.15:4873 , piece_id=X7JNIPBB77LGHIAPSNIS7DB4HMP3SYWDEYMEHPSBNU7D54C2EXOQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.106821505 (579h14m58.97790455s) [noise=true , address=45.91.116.10:4083 , piece_id=HJQOJVJTHV2H46Q5LOY2OJWZHWYFIOM4C5CBPGXTUVHGJZTOT6SQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.107190498 (579h14m58.978273543s) [noise=true , address=145.79.227.51:4411 , piece_id=NGOBFKINEBX4DDFB2IAWKSDTCGAFXSULOFMTC2ZQQ6HVZQM3CS5A, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.107782345 (579h14m58.97886538s) [noise=true , address=145.79.227.4:4185 , piece_id=K2J4GZXJOEPHAIXDE2BJT66XQAMLQCHT7FJGZU54HWYEX3HF5LSA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.108296905 (579h14m58.97937994s) [noise=true , address=145.79.227.26:4654 , piece_id=H6JU7XLOHRRYRDM4QKL4YI4KJVB3PT7J4QWQEWVT3UODZY6LAD4A, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.109437076 (579h14m58.980520111s) [noise=true , address=145.79.227.19:4077 , piece_id=C4C4QVSS4B23QLIY7J2PEZQGADDBAVC26WV4XEYBCGNFZLNCKWTQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.111005072 (579h14m58.982088107s) [noise=true , address=77.90.31.18:4805 , piece_id=7YC63LE3GOLSNZRMARW42GGNIEGMIY6YWEFZ4UZNUONWV2LDXPCQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.112011177 (579h14m58.983094212s) [noise=true , address=45.91.116.6:4975 , piece_id=HOOOKN7U2TS2F6EIWOKE3ZODEPAQFSFCBFJL6CPBC6WJRBB5JFCA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:44:00.806827116 (579h15m0.677910151s) [upload=507782]: Next returned nil segment Dec 29 18:44:00.806845661 (579h15m0.677928696s) [upload=507782]: Waiting for error group managing segments... Dec 29 18:44:01.088383633 (579h15m0.959466678s) [noise=true , address=145.79.227.4:4185 , piece_id=K2J4GZXJOEPHAIXDE2BJT66XQAMLQCHT7FJGZU54HWYEX3HF5LSA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.093613441 (579h15m0.964696486s) [noise=true , address=145.79.227.17:4972 , piece_id=SQ7D3Y6FOA53Z5YRMTBCFIVQZHN6XSNUCWJHLDYIABVQ36KU4CIQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.106339082 (579h15m0.977422127s) [noise=true , address=145.79.227.7:4295 , piece_id=KRYEJUVZF3TCVZERFOIEHPKKEHP36GALEICAWKHI32RS4P5PXCSA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.108290398 (579h15m0.979373443s) [noise=true , address=145.79.227.15:4873 , piece_id=X7JNIPBB77LGHIAPSNIS7DB4HMP3SYWDEYMEHPSBNU7D54C2EXOQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.112039106 (579h15m0.983122151s) [noise=true , address=145.79.227.19:4077 , piece_id=C4C4QVSS4B23QLIY7J2PEZQGADDBAVC26WV4XEYBCGNFZLNCKWTQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.124790687 (579h15m0.995873732s) [noise=true , address=145.79.227.51:4411 , piece_id=NGOBFKINEBX4DDFB2IAWKSDTCGAFXSULOFMTC2ZQQ6HVZQM3CS5A, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.128784301 (579h15m0.999867336s) [noise=true , address=145.79.227.26:4654 , piece_id=H6JU7XLOHRRYRDM4QKL4YI4KJVB3PT7J4QWQEWVT3UODZY6LAD4A, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.318568890 (579h15m1.189651935s) [noise=true , address=77.90.31.11:4555 , piece_id=WLVJH6I7RY4JRSMFIPD6VFE5I37MIR6EH4MSLW5QRBNDYXM6OC5A, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.325917432 (579h15m1.197000477s) [noise=true , address=45.91.116.14:4192 , piece_id=3HWMHIGS7EFFXMSVPSQ2METAEORLEAEBS7JGX5A46DMCCCWE5OIQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.337158277 (579h15m1.208241322s) [noise=true , address=77.90.31.14:4661 , piece_id=FBMFPP3PPBCNSU2IU3I5LOYWFX4EASTPC2VFCXXPSSFIRZP3YISA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.476512488 (579h15m1.347595533s) [noise=true , address=45.91.116.10:4083 , piece_id=HJQOJVJTHV2H46Q5LOY2OJWZHWYFIOM4C5CBPGXTUVHGJZTOT6SQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.965399543 (579h15m1.836482589s) [noise=true , address=77.90.31.18:4805 , piece_id=7YC63LE3GOLSNZRMARW42GGNIEGMIY6YWEFZ4UZNUONWV2LDXPCQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:02.156038740 (579h15m2.027121775s) [noise=true , address=45.91.116.6:4975 , piece_id=HOOOKN7U2TS2F6EIWOKE3ZODEPAQFSFCBFJL6CPBC6WJRBB5JFCA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:02.156064630 (579h15m2.027147675s) [seg_pos={0 0}, upload=507782]: Segment reached optimal threshold of 13 pieces. Dec 29 18:44:02.156324173 (579h15m2.027407248s) [noise=true , address=45.91.116.9:4264 , piece_id=MLD4YWZ53STUN3QW4SSWAA5HPVFX7PFFVC45TL4QFNH4GXBPMADQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36818->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36818->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36818->45.91.116.9:4264: use of closed network connection Dec 29 18:44:02.156334693 (579h15m2.027417738s) [noise=true , address=45.91.116.9:4264 , piece_id=MLD4YWZ53STUN3QW4SSWAA5HPVFX7PFFVC45TL4QFNH4GXBPMADQ, seg_pos={0 0}, upload=507782]: Long tail cancellation due to optimal threshold Dec 29 18:44:02.156573358 (579h15m2.027656403s) [noise=true , address=145.79.227.44:4194 , piece_id=5WX6VRGQNUWG3CYOH46SDEHRMAV6D5SUC6RHCRGAKNUHGBVPRFYA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:02.156581413 (579h15m2.027664458s) [noise=true , address=145.79.227.44:4194 , piece_id=5WX6VRGQNUWG3CYOH46SDEHRMAV6D5SUC6RHCRGAKNUHGBVPRFYA, seg_pos={0 0}, upload=507782]: Long tail cancellation due to optimal threshold Dec 29 18:44:02.156778218 (579h15m2.027861273s) [noise=true , address=145.79.227.35:4482 , piece_id=DZ3KKNC3W7OSMFNAGAONKNILWJFH35C7622RFLP23KZDLXCRFBRA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:02.156807884 (579h15m2.02789093s) [noise=true , address=145.79.227.35:4482 , piece_id=DZ3KKNC3W7OSMFNAGAONKNILWJFH35C7622RFLP23KZDLXCRFBRA, seg_pos={0 0}, upload=507782]: Long tail cancellation due to optimal threshold Dec 29 18:44:02.156867558 (579h15m2.027950603s) [seg_pos={0 0}, upload=507782]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:44:02.157060986 (579h15m2.028144051s) [upload=507782]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:44:02.157087848 (579h15m2.028170893s) [upload=507782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:44:02.470842625 (579h15m2.34192568s) [upload=507782]: Done uploading segments... Dec 29 18:44:02.470871690 (579h15m2.341954735s) [upload=507782]: Upload finished. err: <nil> Duration: 30.00303733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 24.356871532s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:44:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 24.356887992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.430672361s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:44:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430696286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 35.602720844s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:44:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 35.602755368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.469412096s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:43:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469443617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XX] duration: 30.002897192s trace: https://jaeger.dev.storj.io/trace/2f6f2fcdcbdad791 FAIL (2025-12-29 18:43:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 18:42:56.928321377 (579h13m56.799404422s) [upload=507769]: Starting upload Dec 29 18:42:56.928397422 (579h13m56.799480467s) [upload=507769]: Done starting upload Dec 29 18:42:56.928439762 (579h13m56.799522797s) [upload=507769]: Uploading segments... Dec 29 18:42:56.928578917 (579h13m56.799661962s) [upload=507769]: Got next segment. Inline: false Dec 29 18:42:56.928604967 (579h13m56.799688012s) [upload=507769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:42:56.928609205 (579h13m56.79969224s) [upload=507769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:42:57.187127133 (579h13m57.058210178s) [seg_pos={0 0}, upload=507769]: Begin upload segment... Dec 29 18:42:57.187181006 (579h13m57.058264041s) [seg_pos={0 0}, upload=507769]: Done begin upload segment. Dec 29 18:42:57.187224909 (579h13m57.058307954s) [noise=true , address=145.79.227.21:4475 , piece_id=QF7CIKFCLWYQSU6K5LA4Y5UTIVWIVI3FDCVDNBKEA5YSF5V4VLKA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.187246300 (579h13m57.058329335s) [noise=true , address=145.79.227.41:4087 , piece_id=L4FLE732UOZJF35YUM6HB7LJG6ZATWCPN4XHL6ZYH6NK63VNR7WQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.187654306 (579h13m57.058737351s) [noise=true , address=45.91.116.13:4156 , piece_id=DD2XLAIUOU4DOEEGMKX4YQWOITGJ5UDEQBCCLUCSIANAILGSWQPA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188005525 (579h13m57.05908856s) [noise=true , address=145.79.227.11:4730 , piece_id=EKTFZB4SBH7D5BLXXZZVSZ4AOVMN3TBYBBUYH4RR3CYACRODBB7Q, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188051674 (579h13m57.059134709s) [noise=true , address=77.90.31.11:4553 , piece_id=U65ZF7JRESD55A4WO4HPXTYUFEKZLEZHWYQDWTOLDFU63FGVU2DA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188365711 (579h13m57.059448747s) [noise=true , address=145.79.227.10:4402 , piece_id=7XWK3NUMT27776NGXDFFUU3CECOJJGIGFWNTKEM3KKLTW7SUBF3Q, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188409785 (579h13m57.05949282s) [noise=true , address=145.79.227.45:4229 , piece_id=IVPNQTWATZNVWRK7QUE5HCW4RBVSERPYAPVCXYYYVJDTH3EJ4VDQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188674389 (579h13m57.059757424s) [noise=true , address=145.79.227.3:4149 , piece_id=NLDVREIRGGPTAU6ALBTQWFATJSXQ7TGGY4G7BXCJQEJ2JQZ2MNYQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188710759 (579h13m57.059793794s) [noise=true , address=77.90.31.18:4805 , piece_id=477C47D3LV3BBAMFLZX6A2YERH3EXEUBDWYAUE7OIUAMYLHPFQVQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188922882 (579h13m57.060005928s) [noise=true , address=77.90.31.14:4662 , piece_id=JRRFYFW2XV7YZI6COZJM7E7R5HREBHMFNEG2ZPNHGBABW23QTFBQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.189072397 (579h13m57.060155432s) [noise=true , address=145.79.227.6:4257 , piece_id=336JHKXJGANFDK2QLJJGVWBKOCEXYCMGEAEHSFDC7D7GRV5UIHWQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.189434576 (579h13m57.060517621s) [noise=true , address=145.79.227.14:4838 , piece_id=DCZKUNRL4PKXT3FKQRIAGMDSNQ2LDXFM7LVJEL2RLCJNG3WZV6BQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.189874715 (579h13m57.06095776s) [noise=true , address=145.79.227.15:4874 , piece_id=OKGSR57IZ6Z5OA4W3FRHBZEW3NHJWH3C26MLP5GUR46VY7XTGQDA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.190407669 (579h13m57.061490704s) [noise=true , address=145.79.227.29:4942 , piece_id=OG3EUZLYPYBE2EX5QQEEJUMPYOUYARS7XTD726HKHYNDH7ULMEGQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.190834522 (579h13m57.061917557s) [noise=true , address=145.79.227.13:4804 , piece_id=ILSYD6BCJEN5AJSAR6JSDDNN6LBOAJNOS4RCUHJST5ZRTX52CXZA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.191017761 (579h13m57.062100796s) [noise=true , address=145.79.227.46:4601 , piece_id=A7RZLPUTIYSLGF2SQA6A6Z2IUEVQFI277IOEKFQL4FKGNFJX2A5A, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:58.776334415 (579h13m58.64741745s) [upload=507769]: Next returned nil segment Dec 29 18:42:58.776347439 (579h13m58.647430474s) [upload=507769]: Waiting for error group managing segments... Dec 29 18:42:59.157314498 (579h13m59.028397533s) [noise=true , address=145.79.227.15:4874 , piece_id=OKGSR57IZ6Z5OA4W3FRHBZEW3NHJWH3C26MLP5GUR46VY7XTGQDA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.168724575 (579h13m59.03980761s) [noise=true , address=145.79.227.6:4257 , piece_id=336JHKXJGANFDK2QLJJGVWBKOCEXYCMGEAEHSFDC7D7GRV5UIHWQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.172022626 (579h13m59.043105661s) [noise=true , address=145.79.227.45:4229 , piece_id=IVPNQTWATZNVWRK7QUE5HCW4RBVSERPYAPVCXYYYVJDTH3EJ4VDQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.176430539 (579h13m59.047513574s) [noise=true , address=145.79.227.3:4149 , piece_id=NLDVREIRGGPTAU6ALBTQWFATJSXQ7TGGY4G7BXCJQEJ2JQZ2MNYQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.183445145 (579h13m59.05452818s) [noise=true , address=145.79.227.10:4402 , piece_id=7XWK3NUMT27776NGXDFFUU3CECOJJGIGFWNTKEM3KKLTW7SUBF3Q, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.187142356 (579h13m59.058225401s) [noise=true , address=145.79.227.11:4730 , piece_id=EKTFZB4SBH7D5BLXXZZVSZ4AOVMN3TBYBBUYH4RR3CYACRODBB7Q, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.193214358 (579h13m59.064297393s) [noise=true , address=145.79.227.41:4087 , piece_id=L4FLE732UOZJF35YUM6HB7LJG6ZATWCPN4XHL6ZYH6NK63VNR7WQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.196883624 (579h13m59.067966659s) [noise=true , address=145.79.227.13:4804 , piece_id=ILSYD6BCJEN5AJSAR6JSDDNN6LBOAJNOS4RCUHJST5ZRTX52CXZA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.198363872 (579h13m59.069446907s) [noise=true , address=145.79.227.46:4601 , piece_id=A7RZLPUTIYSLGF2SQA6A6Z2IUEVQFI277IOEKFQL4FKGNFJX2A5A, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.199970072 (579h13m59.071053107s) [noise=true , address=145.79.227.21:4475 , piece_id=QF7CIKFCLWYQSU6K5LA4Y5UTIVWIVI3FDCVDNBKEA5YSF5V4VLKA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.209295509 (579h13m59.080378564s) [noise=true , address=145.79.227.14:4838 , piece_id=DCZKUNRL4PKXT3FKQRIAGMDSNQ2LDXFM7LVJEL2RLCJNG3WZV6BQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.337017641 (579h13m59.208100676s) [noise=true , address=77.90.31.11:4553 , piece_id=U65ZF7JRESD55A4WO4HPXTYUFEKZLEZHWYQDWTOLDFU63FGVU2DA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.342283658 (579h13m59.213366693s) [noise=true , address=77.90.31.14:4662 , piece_id=JRRFYFW2XV7YZI6COZJM7E7R5HREBHMFNEG2ZPNHGBABW23QTFBQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.342302364 (579h13m59.213385399s) [seg_pos={0 0}, upload=507769]: Segment reached optimal threshold of 13 pieces. Dec 29 18:42:59.342492215 (579h13m59.21357525s) [noise=true , address=45.91.116.13:4156 , piece_id=DD2XLAIUOU4DOEEGMKX4YQWOITGJ5UDEQBCCLUCSIANAILGSWQPA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 18:42:59.342499268 (579h13m59.213582304s) [noise=true , address=45.91.116.13:4156 , piece_id=DD2XLAIUOU4DOEEGMKX4YQWOITGJ5UDEQBCCLUCSIANAILGSWQPA, seg_pos={0 0}, upload=507769]: Long tail cancellation due to optimal threshold Dec 29 18:42:59.342669533 (579h13m59.213752568s) [noise=true , address=145.79.227.29:4942 , piece_id=OG3EUZLYPYBE2EX5QQEEJUMPYOUYARS7XTD726HKHYNDH7ULMEGQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:42:59.342684923 (579h13m59.213767958s) [noise=true , address=145.79.227.29:4942 , piece_id=OG3EUZLYPYBE2EX5QQEEJUMPYOUYARS7XTD726HKHYNDH7ULMEGQ, seg_pos={0 0}, upload=507769]: Long tail cancellation due to optimal threshold Dec 29 18:42:59.342777298 (579h13m59.213860333s) [noise=true , address=77.90.31.18:4805 , piece_id=477C47D3LV3BBAMFLZX6A2YERH3EXEUBDWYAUE7OIUAMYLHPFQVQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 18:42:59.342785484 (579h13m59.213868519s) [noise=true , address=77.90.31.18:4805 , piece_id=477C47D3LV3BBAMFLZX6A2YERH3EXEUBDWYAUE7OIUAMYLHPFQVQ, seg_pos={0 0}, upload=507769]: Long tail cancellation due to optimal threshold Dec 29 18:42:59.342803498 (579h13m59.213886533s) [seg_pos={0 0}, upload=507769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:42:59.342920962 (579h13m59.214003997s) [upload=507769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:42:59.342927485 (579h13m59.21401052s) [upload=507769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:42:59.656269485 (579h13m59.52735252s) [upload=507769]: Done uploading segments... Dec 29 18:42:59.656290826 (579h13m59.527373861s) [upload=507769]: Upload finished. err: <nil> Duration: 30.002919124s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.421258303s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:43:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.421285525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 30.032753061s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:42:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 30.032769712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 31.333195415s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:42:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 31.333220322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..X] duration: 30.003749728s trace: https://jaeger.dev.storj.io/trace/5d99f245ca2fb9d8 FAIL (2025-12-29 18:42:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 18:41:50.216141708 (579h12m50.087224743s) [upload=507753]: Starting upload Dec 29 18:41:50.216205891 (579h12m50.087288926s) [upload=507753]: Done starting upload Dec 29 18:41:50.216234145 (579h12m50.08731718s) [upload=507753]: Uploading segments... Dec 29 18:41:50.216489611 (579h12m50.087572666s) [upload=507753]: Got next segment. Inline: false Dec 29 18:41:50.216532001 (579h12m50.087615036s) [upload=507753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:41:50.216537021 (579h12m50.087620066s) [upload=507753]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:41:50.463270709 (579h12m50.334353744s) [seg_pos={0 0}, upload=507753]: Begin upload segment... Dec 29 18:41:50.463331064 (579h12m50.33441411s) [seg_pos={0 0}, upload=507753]: Done begin upload segment. Dec 29 18:41:50.463379517 (579h12m50.334462572s) [noise=true , address=77.90.31.18:4808 , piece_id=O2APWAJ7OUPHWV6W2MOETRITDK7SA5CT5HLCZNCQRXVDXN6TEM4A, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.463393704 (579h12m50.334476739s) [noise=true , address=77.90.31.14:4662 , piece_id=KBKIZOXR3PAMMU7SIL66NOZCW6XQCA6C6R4R3WFJ7UTBILSTIYRA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.463446113 (579h12m50.334529158s) [noise=true , address=45.91.116.5:4297 , piece_id=HKJQYPHU4CKAYR23VKJHWP5I4EDFIKHRN3XF3G5JRUDGA7IIYNHQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.464101852 (579h12m50.335184887s) [noise=true , address=45.91.116.14:4191 , piece_id=IXE7DBDK5MGURE4FMK37C6CTLMHN2EKQOLRWVVIJ7CROAW7KZ7RQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.464860847 (579h12m50.335943893s) [noise=true , address=145.79.227.31:4798 , piece_id=BMSVTX6CQFGAJQXGZOJOKPW2TMOUWCMXHJ2TM7GJ63U63Q6Y4UHQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.465471961 (579h12m50.336554996s) [noise=true , address=145.79.227.11:4731 , piece_id=BASLDMVCVW62VRO5GVQ27TAHF2SOKJNSQWV5MLCTLQ6QR72HXQZA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.466352718 (579h12m50.337435763s) [noise=true , address=145.79.227.12:4765 , piece_id=CYADBXI4TA5EJFVOT2C5GQ2VZB5JRM44ODOCYTDXBB7K7K2YAPIQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.466863611 (579h12m50.337946656s) [noise=true , address=145.79.227.43:4159 , piece_id=IDLZIAMIMX4OUQ725GLAOFPCNBG7LRMK3RK6IOHJDABBQBKDIEHA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.468540144 (579h12m50.339623189s) [noise=true , address=145.79.227.13:4803 , piece_id=Z5UGGQBPCL4WB6EVIO3CXDFTDPZPHC7IJYHHV6R7LRPE4QZZR6HA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.470345731 (579h12m50.341428776s) [noise=true , address=45.91.116.8:4047 , piece_id=3PL4TZ6GHSPHKCDCQJ4CNUCDSVHDKIKUZUDMIZQ3BBBHWN2YCFGQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.472160918 (579h12m50.343243953s) [noise=true , address=145.79.227.14:4840 , piece_id=ZCONPUMFMFOXCNCNMBI4CKYKTC3VVIPJLASLKHAG77CKUJRJ6L2Q, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.474434638 (579h12m50.345517673s) [noise=true , address=145.79.227.42:4123 , piece_id=CWUZK47WYZD5WBP5VP7CW2VXTX5SPSNM47IOVZ6IXR5B3BHMDXWQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.476533774 (579h12m50.347616819s) [noise=true , address=45.91.116.12:4228 , piece_id=GE6VNK3UVMLSP5GCMTRYJWEMDKALRPXVTEIUA2UY6QPVV6VAEUJA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.477346172 (579h12m50.348429207s) [noise=true , address=145.79.227.52:4447 , piece_id=VXICGUZSIWWT6G43Y6QHI7IYWLHUNUTIC5ULXVEA2IHTXT6S2NSQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.479081115 (579h12m50.35016415s) [noise=true , address=145.79.227.50:4375 , piece_id=ON7JXLU2VBRC75NC5DU6ZKFEDJ7IQOTAOPWYBC7LPJ5YF4PLVZWA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.479314439 (579h12m50.350397484s) [noise=true , address=145.79.227.16:4912 , piece_id=3OY3DUO3X7CA6B4I7FIQFDLXVWUUD7PGSG4XXXOJ4PVQBVGD65RA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:52.167275495 (579h12m52.03835854s) [upload=507753]: Next returned nil segment Dec 29 18:41:52.167298408 (579h12m52.038381443s) [upload=507753]: Waiting for error group managing segments... Dec 29 18:41:52.458077026 (579h12m52.329160071s) [noise=true , address=145.79.227.31:4798 , piece_id=BMSVTX6CQFGAJQXGZOJOKPW2TMOUWCMXHJ2TM7GJ63U63Q6Y4UHQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.458852071 (579h12m52.329935106s) [noise=true , address=145.79.227.11:4731 , piece_id=BASLDMVCVW62VRO5GVQ27TAHF2SOKJNSQWV5MLCTLQ6QR72HXQZA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.468901526 (579h12m52.339984571s) [noise=true , address=145.79.227.13:4803 , piece_id=Z5UGGQBPCL4WB6EVIO3CXDFTDPZPHC7IJYHHV6R7LRPE4QZZR6HA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.473572330 (579h12m52.344655365s) [noise=true , address=145.79.227.16:4912 , piece_id=3OY3DUO3X7CA6B4I7FIQFDLXVWUUD7PGSG4XXXOJ4PVQBVGD65RA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.481128078 (579h12m52.352211113s) [noise=true , address=145.79.227.50:4375 , piece_id=ON7JXLU2VBRC75NC5DU6ZKFEDJ7IQOTAOPWYBC7LPJ5YF4PLVZWA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.481889758 (579h12m52.352972793s) [noise=true , address=145.79.227.12:4765 , piece_id=CYADBXI4TA5EJFVOT2C5GQ2VZB5JRM44ODOCYTDXBB7K7K2YAPIQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.484012891 (579h12m52.355095926s) [noise=true , address=145.79.227.42:4123 , piece_id=CWUZK47WYZD5WBP5VP7CW2VXTX5SPSNM47IOVZ6IXR5B3BHMDXWQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.493568666 (579h12m52.364651701s) [noise=true , address=145.79.227.14:4840 , piece_id=ZCONPUMFMFOXCNCNMBI4CKYKTC3VVIPJLASLKHAG77CKUJRJ6L2Q, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.498593054 (579h12m52.369676089s) [noise=true , address=145.79.227.52:4447 , piece_id=VXICGUZSIWWT6G43Y6QHI7IYWLHUNUTIC5ULXVEA2IHTXT6S2NSQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.659063574 (579h12m52.530146619s) [noise=true , address=77.90.31.14:4662 , piece_id=KBKIZOXR3PAMMU7SIL66NOZCW6XQCA6C6R4R3WFJ7UTBILSTIYRA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.683381218 (579h12m52.554464263s) [noise=true , address=45.91.116.5:4297 , piece_id=HKJQYPHU4CKAYR23VKJHWP5I4EDFIKHRN3XF3G5JRUDGA7IIYNHQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.694320649 (579h12m52.565403684s) [noise=true , address=77.90.31.18:4808 , piece_id=O2APWAJ7OUPHWV6W2MOETRITDK7SA5CT5HLCZNCQRXVDXN6TEM4A, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.724640933 (579h12m52.595723968s) [noise=true , address=45.91.116.14:4191 , piece_id=IXE7DBDK5MGURE4FMK37C6CTLMHN2EKQOLRWVVIJ7CROAW7KZ7RQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.724665169 (579h12m52.595748204s) [seg_pos={0 0}, upload=507753]: Segment reached optimal threshold of 13 pieces. Dec 29 18:41:52.724903403 (579h12m52.595986448s) [noise=true , address=45.91.116.12:4228 , piece_id=GE6VNK3UVMLSP5GCMTRYJWEMDKALRPXVTEIUA2UY6QPVV6VAEUJA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 18:41:52.724915235 (579h12m52.59599828s) [noise=true , address=45.91.116.12:4228 , piece_id=GE6VNK3UVMLSP5GCMTRYJWEMDKALRPXVTEIUA2UY6QPVV6VAEUJA, seg_pos={0 0}, upload=507753]: Long tail cancellation due to optimal threshold Dec 29 18:41:52.725238331 (579h12m52.596321376s) [noise=true , address=145.79.227.43:4159 , piece_id=IDLZIAMIMX4OUQ725GLAOFPCNBG7LRMK3RK6IOHJDABBQBKDIEHA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 18:41:52.725246577 (579h12m52.596329622s) [noise=true , address=145.79.227.43:4159 , piece_id=IDLZIAMIMX4OUQ725GLAOFPCNBG7LRMK3RK6IOHJDABBQBKDIEHA, seg_pos={0 0}, upload=507753]: Long tail cancellation due to optimal threshold Dec 29 18:41:52.725408354 (579h12m52.596491399s) [noise=true , address=45.91.116.8:4047 , piece_id=3PL4TZ6GHSPHKCDCQJ4CNUCDSVHDKIKUZUDMIZQ3BBBHWN2YCFGQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:37536->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:37536->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37536->45.91.116.8:4047: use of closed network connection Dec 29 18:41:52.725416901 (579h12m52.596499936s) [noise=true , address=45.91.116.8:4047 , piece_id=3PL4TZ6GHSPHKCDCQJ4CNUCDSVHDKIKUZUDMIZQ3BBBHWN2YCFGQ, seg_pos={0 0}, upload=507753]: Long tail cancellation due to optimal threshold Dec 29 18:41:52.725436949 (579h12m52.596519994s) [seg_pos={0 0}, upload=507753]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:41:52.725522482 (579h12m52.596605517s) [upload=507753]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:41:52.725528644 (579h12m52.596611689s) [upload=507753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:41:53.015240038 (579h12m52.886323083s) [upload=507753]: Done uploading segments... Dec 29 18:41:53.015268191 (579h12m52.886351237s) [upload=507753]: Upload finished. err: <nil> Duration: 30.003770587s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.530089284s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:42:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530113632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 27.341021093s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:41:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 27.34105069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..] duration: 5.959283283s trace: https://jaeger.dev.storj.io/trace/205c371f3e3d99d5 FAIL (2025-12-29 18:41:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA) context deadline exceeded ======LOGS====== Dec 29 18:39:05.269762644 (579h10m5.140845679s) [upload=507714]: Starting upload Dec 29 18:39:05.269842036 (579h10m5.140925081s) [upload=507714]: Done starting upload Dec 29 18:39:05.270022539 (579h10m5.141105574s) [upload=507714]: Uploading segments... Dec 29 18:39:05.270072644 (579h10m5.141155689s) [upload=507714]: Got next segment. Inline: false Dec 29 18:39:05.270102702 (579h10m5.141185747s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:39:05.270108883 (579h10m5.141191928s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:39:05.516713306 (579h10m5.387796351s) [seg_pos={0 0}, upload=507714]: Begin upload segment... Dec 29 18:39:05.516761808 (579h10m5.387844853s) [seg_pos={0 0}, upload=507714]: Done begin upload segment. Dec 29 18:39:05.516793349 (579h10m5.387876384s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516801073 (579h10m5.387884108s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516848785 (579h10m5.38793182s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517226514 (579h10m5.388309559s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517508631 (579h10m5.388591676s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517767684 (579h10m5.388850729s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518158778 (579h10m5.389241813s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518496552 (579h10m5.389579597s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518910821 (579h10m5.389993866s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519333395 (579h10m5.39041643s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519765489 (579h10m5.390848534s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520534152 (579h10m5.391617187s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520957829 (579h10m5.392040874s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.521587858 (579h10m5.392670903s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.523484429 (579h10m5.394567464s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.524676169 (579h10m5.395759214s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:07.233782657 (579h10m7.104865692s) [upload=507714]: Next returned nil segment Dec 29 18:39:07.233825318 (579h10m7.104908353s) [upload=507714]: Waiting for error group managing segments... Dec 29 18:39:07.521013829 (579h10m7.392096874s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.521890689 (579h10m7.392973734s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.525031880 (579h10m7.396114925s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.543925366 (579h10m7.415008411s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.559780046 (579h10m7.430863091s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.740401066 (579h10m7.611484101s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.853332184 (579h10m7.724415229s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.897665411 (579h10m7.768748476s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.951662390 (579h10m7.822745435s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.506526966 (579h10m8.377610011s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.843235771 (579h10m8.714318806s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:09.174697294 (579h10m9.045780339s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635844411 (579h10m10.506927456s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635868427 (579h10m10.506951472s) [seg_pos={0 0}, upload=507714]: Segment reached optimal threshold of 13 pieces. Dec 29 18:39:10.636079248 (579h10m10.507162283s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636086802 (579h10m10.507169848s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636202763 (579h10m10.507285808s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636223914 (579h10m10.507306949s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636256365 (579h10m10.507339401s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636264411 (579h10m10.507347456s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636291773 (579h10m10.507385538s) [seg_pos={0 0}, upload=507714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:39:10.636416190 (579h10m10.507499225s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:39:10.636422712 (579h10m10.507505748s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:39:10.932641628 (579h10m10.803724673s) [upload=507714]: Done uploading segments... Dec 29 18:39:10.932664472 (579h10m10.803747507s) [upload=507714]: Upload finished. err: <nil> Duration: 5.959315134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 26.575404575s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:40:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 26.575432689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.516992621s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:40:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517021045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.4150689s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:40:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415094628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X.] duration: 6.090212775s trace: https://jaeger.dev.storj.io/trace/205c371f3e3d99d5 FAIL (2025-12-29 18:40:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA) context deadline exceeded ======LOGS====== Dec 29 18:39:05.269762644 (579h10m5.140845679s) [upload=507714]: Starting upload Dec 29 18:39:05.269842036 (579h10m5.140925081s) [upload=507714]: Done starting upload Dec 29 18:39:05.270022539 (579h10m5.141105574s) [upload=507714]: Uploading segments... Dec 29 18:39:05.270072644 (579h10m5.141155689s) [upload=507714]: Got next segment. Inline: false Dec 29 18:39:05.270102702 (579h10m5.141185747s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:39:05.270108883 (579h10m5.141191928s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:39:05.516713306 (579h10m5.387796351s) [seg_pos={0 0}, upload=507714]: Begin upload segment... Dec 29 18:39:05.516761808 (579h10m5.387844853s) [seg_pos={0 0}, upload=507714]: Done begin upload segment. Dec 29 18:39:05.516793349 (579h10m5.387876384s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516801073 (579h10m5.387884108s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516848785 (579h10m5.38793182s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517226514 (579h10m5.388309559s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517508631 (579h10m5.388591676s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517767684 (579h10m5.388850729s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518158778 (579h10m5.389241813s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518496552 (579h10m5.389579597s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518910821 (579h10m5.389993866s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519333395 (579h10m5.39041643s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519765489 (579h10m5.390848534s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520534152 (579h10m5.391617187s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520957829 (579h10m5.392040874s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.521587858 (579h10m5.392670903s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.523484429 (579h10m5.394567464s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.524676169 (579h10m5.395759214s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:07.233782657 (579h10m7.104865692s) [upload=507714]: Next returned nil segment Dec 29 18:39:07.233825318 (579h10m7.104908353s) [upload=507714]: Waiting for error group managing segments... Dec 29 18:39:07.521013829 (579h10m7.392096874s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.521890689 (579h10m7.392973734s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.525031880 (579h10m7.396114925s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.543925366 (579h10m7.415008411s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.559780046 (579h10m7.430863091s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.740401066 (579h10m7.611484101s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.853332184 (579h10m7.724415229s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.897665411 (579h10m7.768748476s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.951662390 (579h10m7.822745435s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.506526966 (579h10m8.377610011s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.843235771 (579h10m8.714318806s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:09.174697294 (579h10m9.045780339s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635844411 (579h10m10.506927456s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635868427 (579h10m10.506951472s) [seg_pos={0 0}, upload=507714]: Segment reached optimal threshold of 13 pieces. Dec 29 18:39:10.636079248 (579h10m10.507162283s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636086802 (579h10m10.507169848s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636202763 (579h10m10.507285808s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636223914 (579h10m10.507306949s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636256365 (579h10m10.507339401s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636264411 (579h10m10.507347456s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636291773 (579h10m10.507385538s) [seg_pos={0 0}, upload=507714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:39:10.636416190 (579h10m10.507499225s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:39:10.636422712 (579h10m10.507505748s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:39:10.932641628 (579h10m10.803724673s) [upload=507714]: Done uploading segments... Dec 29 18:39:10.932664472 (579h10m10.803747507s) [upload=507714]: Upload finished. err: <nil> Duration: 6.090244705s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.479477648s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:40:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479632282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.41220763s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:39:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.412237277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X] duration: 30.003213554s trace: https://jaeger.dev.storj.io/trace/205c371f3e3d99d5 FAIL (2025-12-29 18:39:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA) context deadline exceeded ======LOGS====== Dec 29 18:39:05.269762644 (579h10m5.140845679s) [upload=507714]: Starting upload Dec 29 18:39:05.269842036 (579h10m5.140925081s) [upload=507714]: Done starting upload Dec 29 18:39:05.270022539 (579h10m5.141105574s) [upload=507714]: Uploading segments... Dec 29 18:39:05.270072644 (579h10m5.141155689s) [upload=507714]: Got next segment. Inline: false Dec 29 18:39:05.270102702 (579h10m5.141185747s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:39:05.270108883 (579h10m5.141191928s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:39:05.516713306 (579h10m5.387796351s) [seg_pos={0 0}, upload=507714]: Begin upload segment... Dec 29 18:39:05.516761808 (579h10m5.387844853s) [seg_pos={0 0}, upload=507714]: Done begin upload segment. Dec 29 18:39:05.516793349 (579h10m5.387876384s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516801073 (579h10m5.387884108s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516848785 (579h10m5.38793182s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517226514 (579h10m5.388309559s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517508631 (579h10m5.388591676s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517767684 (579h10m5.388850729s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518158778 (579h10m5.389241813s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518496552 (579h10m5.389579597s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518910821 (579h10m5.389993866s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519333395 (579h10m5.39041643s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519765489 (579h10m5.390848534s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520534152 (579h10m5.391617187s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520957829 (579h10m5.392040874s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.521587858 (579h10m5.392670903s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.523484429 (579h10m5.394567464s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.524676169 (579h10m5.395759214s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:07.233782657 (579h10m7.104865692s) [upload=507714]: Next returned nil segment Dec 29 18:39:07.233825318 (579h10m7.104908353s) [upload=507714]: Waiting for error group managing segments... Dec 29 18:39:07.521013829 (579h10m7.392096874s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.521890689 (579h10m7.392973734s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.525031880 (579h10m7.396114925s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.543925366 (579h10m7.415008411s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.559780046 (579h10m7.430863091s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.740401066 (579h10m7.611484101s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.853332184 (579h10m7.724415229s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.897665411 (579h10m7.768748476s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.951662390 (579h10m7.822745435s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.506526966 (579h10m8.377610011s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.843235771 (579h10m8.714318806s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:09.174697294 (579h10m9.045780339s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635844411 (579h10m10.506927456s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635868427 (579h10m10.506951472s) [seg_pos={0 0}, upload=507714]: Segment reached optimal threshold of 13 pieces. Dec 29 18:39:10.636079248 (579h10m10.507162283s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636086802 (579h10m10.507169848s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636202763 (579h10m10.507285808s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636223914 (579h10m10.507306949s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636256365 (579h10m10.507339401s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636264411 (579h10m10.507347456s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636291773 (579h10m10.507385538s) [seg_pos={0 0}, upload=507714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:39:10.636416190 (579h10m10.507499225s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:39:10.636422712 (579h10m10.507505748s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:39:10.932641628 (579h10m10.803724673s) [upload=507714]: Done uploading segments... Dec 29 18:39:10.932664472 (579h10m10.803747507s) [upload=507714]: Upload finished. err: <nil> Duration: 30.00324277s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.584454477s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:39:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.584486648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.39453502s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:38:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.394566089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 35.825453372s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:38:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 35.825480924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......] duration: 9.842411349s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:38:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 9.842431256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX......] duration: 6.025738586s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:38:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 6.025797208s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.641385802s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:37:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.641412023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 25.044123909s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:37:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 25.044138617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 30.290809867s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:37:19 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 30.290836047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.....] duration: 6.048956866s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:37:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 6.048997374s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.549750705s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:36:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549777536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 22.181070058s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:36:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 22.181089756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX....] duration: 6.062221592s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 6.062250307s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)