Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.....................X........] duration: 11.350064393s trace: https://jaeger.dev.storj.io/trace/2a592caa2cd77abc FAIL (2026-01-03 13:00:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.350091835s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................X.X........X.............X.................] duration: 28.303986865s trace: https://jaeger.dev.storj.io/trace/165637e999409e3 FAIL (2026-01-03 13:00:51 UTC) orbiter warning: context deadline exceeded Duration: 28.304230899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 10 out of 60 is failed (threshold=3) [...X..X.....X...X................X.....X..X...X.X......X....] duration: 9.458019407s trace: https://jaeger.dev.storj.io/trace/53bc30794fd42896 FAIL (2026-01-03 13:00:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 22) ======LOGS====== 286]: Uploading piece... Jan 3 12:57:19.136747473 (693h28m19.007830508s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137034108 (693h28m19.008117143s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137189855 (693h28m19.008272891s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137413200 (693h28m19.008496245s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137560532 (693h28m19.008643567s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137902893 (693h28m19.008985938s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138193637 (693h28m19.009276692s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138313765 (693h28m19.0093968s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138696524 (693h28m19.009779569s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:20.027400153 (693h28m19.898483188s) [upload=607286]: Next returned nil segment Jan 3 12:57:20.027453063 (693h28m19.898536108s) [upload=607286]: Waiting for error group managing segments... Jan 3 12:57:20.410138690 (693h28m20.281221725s) [noise=true , address=185.167.204.205:4002 , piece_id=MOWRJZ3WLWZZXXB5ENPBOVZGVAD2V353VZ5CWCEO22PPZQGROVHQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:20.492391803 (693h28m20.363474848s) [noise=true , address=185.167.204.205:4022 , piece_id=CEGF62VNP7MZO7WADY2WL7WZX6UDOUA5NTWGGODTJGNKHBCCGJVA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.019412959 (693h28m20.890496004s) [noise=true , address=81.171.14.70:4030 , piece_id=GDSHOILKCEFT5BP2R6ZZDUQ5DF4CO5H3OD754ROW6GFRU3UU35TA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.054989251 (693h28m20.926072287s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.057129006 (693h28m20.928212052s) [noise=true , address=81.171.14.70:4028 , piece_id=P42SQIYXEJJKQEI2GFXRU5JIJ5G3XHLT2SM45DCE67E7S5TSPSYA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.060624582 (693h28m20.931707617s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.066618325 (693h28m20.93770136s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.076354184 (693h28m20.947437219s) [noise=true , address=81.171.14.70:4046 , piece_id=CXZXC744XXEYRZ252W2JM5N3VXFWA6UHUQYM6Z5WME7Q2B66YS5Q, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.084082891 (693h28m20.955165936s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.565324133 (693h28m21.436407168s) [noise=true , address=81.171.14.70:4032 , piece_id=5X2IPF6QMSEKVGMZFXBBW3DVXAK5SD3ZELAGWPRFN5C6SEVNUNWA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.584630255 (693h28m21.455713291s) [noise=true , address=81.171.14.70:4038 , piece_id=JI4QF46ZDL3TM4EFESCJSFFKIYRTF7CJZCC2HMUQZGO4CP2M4V5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.587294258 (693h28m21.458377303s) [noise=true , address=81.171.14.70:4034 , piece_id=GAFESRFTUPAOIHJ5LKEFK7AIZMLKL7JO5NCC26ZR27PQTTYH5UUA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.874175286 (693h28m21.745258321s) [noise=true , address=89.149.201.147:4016 , piece_id=MA3LBUYQ6OY7E4HMSROO5FVKJE7CEXF7SXUJIWWQX4EQCLI3OC7A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.284508037 (693h28m22.155591072s) [noise=true , address=89.149.201.147:4006 , piece_id=TQWVNQARGBZR2SRZ2LA3EVXSASEX36WXIEBGXWNWH2VQDSVC6DTA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.378387289 (693h28m22.249470334s) [noise=true , address=89.149.201.147:4015 , piece_id=SMHKGOSXQUWCO24KJLOH5RTLTZJ7MOADSCGPC6WRZUQFQHNNQ6WQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:23.055399573 (693h28m22.926482618s) [noise=true , address=81.171.14.70:4043 , piece_id=XKAAVOXN772VX5A6HSKBOU5NOYYF4U3HZ2BA5PYEOOQKCFGAIAFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:28.963038821 (693h28m28.834121857s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:32.765601718 (693h28m32.636684753s) [noise=true , address=185.167.204.205:4019 , piece_id=JKTSOUCJ2OF43PZKOKLRGUHSKRFWHQNMTIESGBK4MQNZO6VTWCLQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:36.941962859 (693h28m36.813045904s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:40.179862153 (693h28m40.050945188s) [noise=true , address=185.167.204.205:4010 , piece_id=CJBQASNMJAA5VBAFF65GU22ITWXU3RLHF6B7TFIDSUWCGSJM3OIA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:43.522593496 (693h28m43.393676531s) [noise=true , address=185.167.204.206:4041 , piece_id=QVVTZZDAAESA7IKRLWUMFMXOM4WIRUA6DP4FU2YKXSLK4WPTEC4A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:48.895462434 (693h28m48.766545479s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:58:18.486768101 (693h29m18.357851136s) [noise=true , address=185.167.204.205:4025 , piece_id=OFNEI5EOXKHZNLNDSJUQSMOBYUIGCFSAVLKRSJ5TKB3KX6ZXGN5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1cnhWXZCgZz567E5PAaPyfNKkmsSa4imaWPmfW2ir7U48gYEkp, address:185.167.204.205:4025): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487034448 (693h29m18.358117483s) [noise=true , address=185.167.204.206:4047 , piece_id=ZHLGMV427BQKUCYXOXRSMGVYTQH5TR7H62BKMOHY6QZHFQLCY2BQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487255509 (693h29m18.358338554s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12hK2MSVcFNaCVp7k4krUva7jXEJYhjnHPxPrMgq87jcDXJPkv2, address:185.167.204.206:4051): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487444569 (693h29m18.358527604s) [noise=true , address=185.167.204.205:4027 , piece_id=VIHPQE3Q7CRTAVPIYKTT4PIRU3RW4LPLUUWULRT66IX574KZ2GBA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487563535 (693h29m18.35864657s) [noise=true , address=185.167.204.205:4011 , piece_id=QBDCX7YUNQXSNX5BNO62KY4EZF7Y24KSMGDVR3XMBZF4JVM4C5FQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: read tcp 188.166.225.170:40296->185.167.204.205:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487627026 (693h29m18.358710072s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: read tcp 188.166.225.170:45010->185.167.204.205:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487692902 (693h29m18.358775937s) [seg_pos={0 0}, upload=607286]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil> Jan 3 12:58:18.668736929 (693h29m18.539819974s) [upload=607286]: Done uploading segments... Jan 3 12:58:18.668774270 (693h29m18.539857315s) [upload=607286]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22) Duration: 9.458042802s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXX.X.XXXXXXX.XXXX.XXX.XX.XX....X.XXXXXXX.X] duration: 5.001707605s trace: https://jaeger.dev.storj.io/trace/40c5e6ce0c431a55 FAIL (2026-01-03 13:00:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001729867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X..X.XX.X.....XXX.XXXX..X..X...X..XX..XX..X...XX..XX.X....] duration: 5.685483688s trace: https://jaeger.dev.storj.io/trace/6384b51b065162e6 FAIL (2026-01-03 13:00:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: 4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA) context deadline exceeded; piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: 3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA) context deadline exceeded ======LOGS====== Jan 3 12:57:32.139975894 (693h28m32.011058929s) [upload=607290]: Starting upload Jan 3 12:57:32.140035398 (693h28m32.011118433s) [upload=607290]: Done starting upload Jan 3 12:57:32.140163461 (693h28m32.011246496s) [upload=607290]: Uploading segments... Jan 3 12:57:32.140286756 (693h28m32.011369801s) [upload=607290]: Got next segment. Inline: false Jan 3 12:57:32.140334296 (693h28m32.011417331s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 3 12:57:32.140339225 (693h28m32.01142226s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 3 12:57:32.386239220 (693h28m32.257322265s) [seg_pos={0 0}, upload=607290]: Begin upload segment... Jan 3 12:57:32.386291079 (693h28m32.257374114s) [seg_pos={0 0}, upload=607290]: Done begin upload segment. Jan 3 12:57:32.386337226 (693h28m32.257420261s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386347246 (693h28m32.257430281s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386399996 (693h28m32.257483041s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386793715 (693h28m32.257876751s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387239204 (693h28m32.258322239s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387746029 (693h28m32.258829065s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.388247554 (693h28m32.259330619s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.390590996 (693h28m32.261674041s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.394182876 (693h28m32.265265912s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.397175685 (693h28m32.26825872s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.398532489 (693h28m32.269615534s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.400607720 (693h28m32.271690755s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.402897280 (693h28m32.273980315s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.403308622 (693h28m32.274391657s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.405982995 (693h28m32.27706603s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.407419360 (693h28m32.278502395s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:34.297376941 (693h28m34.168459976s) [upload=607290]: Next returned nil segment Jan 3 12:57:34.297397681 (693h28m34.168480716s) [upload=607290]: Waiting for error group managing segments... Jan 3 12:57:34.931863919 (693h28m34.802946954s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.117922626 (693h28m34.989005671s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.556098205 (693h28m35.42718125s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.595938106 (693h28m35.467021141s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.211248298 (693h28m36.082331333s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.430239493 (693h28m36.301322528s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.787016996 (693h28m36.658100031s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.437957986 (693h28m37.309041031s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.960941243 (693h28m37.832024278s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:38.634528192 (693h28m38.505611227s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.241920683 (693h28m40.113003718s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.826824296 (693h28m40.697907331s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612734057 (693h28m41.483817092s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612755909 (693h28m41.483838954s) [seg_pos={0 0}, upload=607290]: Segment reached optimal threshold of 13 pieces. Jan 3 12:57:41.613062503 (693h28m41.484145548s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: read tcp 188.166.225.170:48988->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:57:41.613069997 (693h28m41.484153032s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613316457 (693h28m41.484399492s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613324452 (693h28m41.484407487s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613457245 (693h28m41.48454029s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613494676 (693h28m41.484577731s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613539030 (693h28m41.484622075s) [seg_pos={0 0}, upload=607290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 3 12:57:41.613773797 (693h28m41.484856842s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:57:41.613782373 (693h28m41.484865428s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:57:41.908933460 (693h28m41.780016505s) [upload=607290]: Done uploading segments... Jan 3 12:57:41.908950523 (693h28m41.780033558s) [upload=607290]: Upload finished. err: <nil> Duration: 5.68551627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........X.................X......................] duration: 9.330613867s trace: https://jaeger.dev.storj.io/trace/640fd8ac8dea8303 FAIL (2026-01-03 13:00:38 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 9.330638353s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [...X.XX....XX.XXX.XXX.XXXX.XXXXX.XX.XXXXXX.XX.XXX.X.XX.X.XXX] duration: 30.001402787s trace: https://jaeger.dev.storj.io/trace/3a46fc5a49954668 FAIL (2026-01-03 13:00:38 UTC) orbiter warning: context deadline exceeded Duration: 30.001437954s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXX.X...X...XX.X.X.XXX.XXXXX..XX...X.X.X..XX.XXXX] duration: 1m0.005601515s trace: https://jaeger.dev.storj.io/trace/7d7328b408d31528 FAIL (2026-01-03 13:00:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ODTGA55EIEPDGHFK5IVY5NSLTQPFXY32THYTY5Q, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.707793048 (693h30m34.578876083s) [noise=true , address=145.79.227.43:4166 , piece_id=KFYGHFI7ZFIF2VA4GNEAAKND4SPQEVXR4PHDRGOR7AUGOJNR4GXQ, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.708143215 (693h30m34.57922626s) [noise=true , address=77.90.31.11:4587 , piece_id=U2OOSXOKR4UOIC5XLDXNPELRINLKO4NIFJTF6SX73FUN2Z5WSTYA, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.708650139 (693h30m34.579733175s) [noise=true , address=145.79.227.30:4779 , piece_id=P5IVZ6N5XA2B7N6Q7KLC4VK47SCRQEIBHU4MODB6SNV62JPLYPBQ, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.708744650 (693h30m34.579827685s) [noise=true , address=145.79.227.40:4083 , piece_id=PYYVXVP6PP5BXCNYOQSJXD3BYG4JJAMUAFLG4D3RDD7YVGCYA6WA, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.709742350 (693h30m34.580825395s) [noise=true , address=145.79.227.33:4898 , piece_id=ZABSFIKYVDI7Z6C622BK4EKWX6GJ656WDRW423QZJNX2OIQFEQ3A, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.709850175 (693h30m34.58093321s) [noise=true , address=77.90.31.15:4727 , piece_id=XHNPF4ADH7JLVDXQBWLTIXICDUHGLI3AYZSHWARNJ6DGRVIX6IFA, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.710796246 (693h30m34.581879291s) [noise=true , address=45.91.116.5:4307 , piece_id=ZDB4PCFCAREUG4ZYRESCBHO7OHPBDUYMUV27ZG5XTYAEZGEM5BOQ, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.710868814 (693h30m34.58195185s) [noise=true , address=145.79.227.29:4973 , piece_id=A4W7354QUBMQ45525T7URZII2KLMKWKKLSJYKYPOM5AKMFKHYBBQ, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.711476391 (693h30m34.582559426s) [noise=true , address=77.90.31.14:4670 , piece_id=GYK2UIBXKXJ46KGNXAFAZPXFZDQCGHHBQIDHS7JB3MQOJFA2KSYA, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.711832350 (693h30m34.582915385s) [noise=true , address=45.91.116.10:4103 , piece_id=FCHNUTRRAIQGPZIKDNFYLE7VHJGOYBR3PEIMN4BCPJIOJ4HVQCVQ, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.712188297 (693h30m34.583271332s) [noise=true , address=145.79.227.38:4984 , piece_id=MQHYHTE5ZYQB7QG6MWDON2YXBYK5WWA6KSUQZKZ3AURIJOYIBP6A, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.712451458 (693h30m34.583534493s) [noise=true , address=45.91.116.11:4141 , piece_id=XJ6DX7HIMRP6PWFQACTTB6PQRSQRJOJJ5FR56OAYT7ACLKYHIWHQ, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.712532853 (693h30m34.583615888s) [noise=true , address=77.90.31.18:4839 , piece_id=SHLRRO3L4OO2UZY6764WNPRIQDHC2F335VGHADMOHBY56ZHAGRQQ, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.712864785 (693h30m34.58394782s) [noise=true , address=77.90.31.16:4746 , piece_id=5OFR3AIYN5FDRE7EQQTYNSFMCWYFFQV5WDCDY2QYK327VUL3F76A, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:34.712948535 (693h30m34.58403157s) [noise=true , address=77.90.31.13:4647 , piece_id=FTVAAUMMUNQU23EVLTBQ3HD4E746AWA2PFVLKA6HG3EFBCQOHRBA, seg_pos={0 0}, upload=607319]: Uploading piece... Jan 3 12:59:36.174304347 (693h30m36.045387392s) [upload=607319]: Next returned nil segment Jan 3 12:59:36.174318704 (693h30m36.045401749s) [upload=607319]: Waiting for error group managing segments... Jan 3 12:59:36.588962143 (693h30m36.460045188s) [noise=true , address=145.79.227.38:4984 , piece_id=MQHYHTE5ZYQB7QG6MWDON2YXBYK5WWA6KSUQZKZ3AURIJOYIBP6A, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.084596872 (693h30m36.955679917s) [noise=true , address=77.90.31.15:4727 , piece_id=XHNPF4ADH7JLVDXQBWLTIXICDUHGLI3AYZSHWARNJ6DGRVIX6IFA, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.100264867 (693h30m36.971347903s) [noise=true , address=45.91.116.11:4141 , piece_id=XJ6DX7HIMRP6PWFQACTTB6PQRSQRJOJJ5FR56OAYT7ACLKYHIWHQ, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.245523513 (693h30m37.116606558s) [noise=true , address=45.91.116.10:4103 , piece_id=FCHNUTRRAIQGPZIKDNFYLE7VHJGOYBR3PEIMN4BCPJIOJ4HVQCVQ, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.425948192 (693h30m37.297031227s) [noise=true , address=145.79.227.44:4210 , piece_id=5ECI4ZFMPYR5WY6DVUF4HGWRU5RFWUEKNS5LL3ZR5YCAQ6BFHVOQ, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.498801817 (693h30m37.369884862s) [noise=true , address=145.79.227.29:4973 , piece_id=A4W7354QUBMQ45525T7URZII2KLMKWKKLSJYKYPOM5AKMFKHYBBQ, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.500294499 (693h30m37.371377544s) [noise=true , address=145.79.227.35:4489 , piece_id=3KHMYQRYF4LHOODTGA55EIEPDGHFK5IVY5NSLTQPFXY32THYTY5Q, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.506063744 (693h30m37.377146789s) [noise=true , address=145.79.227.33:4898 , piece_id=ZABSFIKYVDI7Z6C622BK4EKWX6GJ656WDRW423QZJNX2OIQFEQ3A, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.523521588 (693h30m37.394604633s) [noise=true , address=145.79.227.30:4779 , piece_id=P5IVZ6N5XA2B7N6Q7KLC4VK47SCRQEIBHU4MODB6SNV62JPLYPBQ, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.604410042 (693h30m37.475493088s) [noise=true , address=145.79.227.40:4083 , piece_id=PYYVXVP6PP5BXCNYOQSJXD3BYG4JJAMUAFLG4D3RDD7YVGCYA6WA, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.610606682 (693h30m37.481689727s) [noise=true , address=145.79.227.43:4166 , piece_id=KFYGHFI7ZFIF2VA4GNEAAKND4SPQEVXR4PHDRGOR7AUGOJNR4GXQ, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:37.783262167 (693h30m37.654345212s) [noise=true , address=145.79.227.34:4940 , piece_id=3ZOSMK33IG2NZOPOGLL4LH5SHAEMKXJZLRJWCYWMRJGES7EXMTRA, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:39.933053935 (693h30m39.80413698s) [noise=true , address=45.91.116.5:4307 , piece_id=ZDB4PCFCAREUG4ZYRESCBHO7OHPBDUYMUV27ZG5XTYAEZGEM5BOQ, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:40.539485260 (693h30m40.410568295s) [noise=true , address=77.90.31.11:4587 , piece_id=U2OOSXOKR4UOIC5XLDXNPELRINLKO4NIFJTF6SX73FUN2Z5WSTYA, seg_pos={0 0}, upload=607319]: Done uploading piece. err: <nil> Jan 3 12:59:40.539517191 (693h30m40.410600246s) [seg_pos={0 0}, upload=607319]: Segment reached optimal threshold of 14 pieces. Jan 3 12:59:40.539893598 (693h30m40.410976633s) [noise=true , address=77.90.31.16:4746 , piece_id=5OFR3AIYN5FDRE7EQQTYNSFMCWYFFQV5WDCDY2QYK327VUL3F76A, seg_pos={0 0}, upload=607319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: read tcp 188.166.225.170:54650->77.90.31.16:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:59:40.539903717 (693h30m40.410986762s) [noise=true , address=77.90.31.16:4746 , piece_id=5OFR3AIYN5FDRE7EQQTYNSFMCWYFFQV5WDCDY2QYK327VUL3F76A, seg_pos={0 0}, upload=607319]: Long tail cancellation due to optimal threshold Jan 3 12:59:40.539925228 (693h30m40.411008263s) [noise=true , address=77.90.31.13:4647 , piece_id=FTVAAUMMUNQU23EVLTBQ3HD4E746AWA2PFVLKA6HG3EFBCQOHRBA, seg_pos={0 0}, upload=607319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv): protocol: read tcp 188.166.225.170:34756->77.90.31.13:4647: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:59:40.539938914 (693h30m40.411021949s) [noise=true , address=77.90.31.13:4647 , piece_id=FTVAAUMMUNQU23EVLTBQ3HD4E746AWA2PFVLKA6HG3EFBCQOHRBA, seg_pos={0 0}, upload=607319]: Long tail cancellation due to optimal threshold Jan 3 12:59:40.540066948 (693h30m40.411149993s) [noise=true , address=77.90.31.14:4670 , piece_id=GYK2UIBXKXJ46KGNXAFAZPXFZDQCGHHBQIDHS7JB3MQOJFA2KSYA, seg_pos={0 0}, upload=607319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LQ2JYC2GB67iqFPW69DyASSKH9yN558jGYBFd2cRN8abegiTE): protocol: expected piece hash; context canceled; EOF Jan 3 12:59:40.540073811 (693h30m40.411156856s) [noise=true , address=77.90.31.14:4670 , piece_id=GYK2UIBXKXJ46KGNXAFAZPXFZDQCGHHBQIDHS7JB3MQOJFA2KSYA, seg_pos={0 0}, upload=607319]: Long tail cancellation due to optimal threshold Jan 3 12:59:40.540207526 (693h30m40.411290561s) [noise=true , address=77.90.31.18:4839 , piece_id=SHLRRO3L4OO2UZY6764WNPRIQDHC2F335VGHADMOHBY56ZHAGRQQ, seg_pos={0 0}, upload=607319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: read tcp 188.166.225.170:57462->77.90.31.18:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:59:40.540215671 (693h30m40.411298716s) [noise=true , address=77.90.31.18:4839 , piece_id=SHLRRO3L4OO2UZY6764WNPRIQDHC2F335VGHADMOHBY56ZHAGRQQ, seg_pos={0 0}, upload=607319]: Long tail cancellation due to optimal threshold Jan 3 12:59:40.540238024 (693h30m40.411321059s) [seg_pos={0 0}, upload=607319]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 3 12:59:40.540341450 (693h30m40.411424495s) [upload=607319]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:59:40.540348204 (693h30m40.411431239s) [upload=607319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:59:40.833503600 (693h30m40.704586645s) [upload=607319]: Done uploading segments... Jan 3 12:59:40.833519651 (693h30m40.704602696s) [upload=607319]: Upload finished. err: <nil> Duration: 1m0.005618648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.321080069s trace: https://jaeger.dev.storj.io/trace/5a0aed6ab18c8274 FAIL (2026-01-03 13:00:32 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 1F3yVaqH52tef8gWWe4JAWh47yRjDDEqYyou7abzPnDkDEeson, Piece ID: FEJXLLJSEBSH4XFEJX54FSB3U3237HMWQWZJAIGHDRFQHPNB5KLA) context deadline exceeded; piecestore: (Node ID: 1yyv7LXhZGJFvkTV4vafuZ6KrQkBWB7zH7q2emQ3dNat8iuWsK, Piece ID: OWDX5T5OCZF6L7DTGORIV24KCTBVMI5NUWJC5JZVGYOR36KS3DPQ) context deadline exceeded; piecestore: (Node ID: 12q1oXSNGG6uCuHBtvGTxrbWLnchY11hmhfbiRzcm1zCwGvz7Zh, Piece ID: T4E3TYY6CEPW54CARC2EIJS6YF4CDABBI7LIJQX5Q6AVFO7JEVNQ) context deadline exceeded; piecestore: (Node ID: 1UtnhDdcRHT4L5h1MxGCRWfY24bCETpRgMz62K8s4qAMkwS3fG, Piece ID: 64IAPWLKLFHF7MI3IAHFZHBNOTH64G4OPBC2NGPL5DZCWJHBXOFQ) context deadline exceeded; piecestore: (Node ID: 1x29ZBapLNUJt9E9WjyjUYav5LK4d3h5MsmZAVH2L2daqiANw1, Piece ID: QPZFZ3N4E2VSRMUB7XU6LZUH4WJFVEFYQMLTZ6O7ACG5RMBHNDJA) context deadline exceeded; piecestore: (Node ID: 1EQTE1zDuMWpoXmTB2jkH9cv2Pe2yAGWCc6A4iNznkSKzkN8pM, Piece ID: SAYUQDKORTF7IF7XHIUXCI454RUHCCBQPCIBKMX6T2E6LR2DRRUA) context deadline exceeded; piecestore: (Node ID: 1o2qgqfNpCnXxd66kmxTTqeQzWzuegmzrhqkP9bZ9CXmjLzva5, Piece ID: Q5AR3ZKTR6TX2ZQAH34DDVLT7IP5Y5N4HKSHNJPJQMS6QNTRGXIA) context deadline exceeded; piecestore: (Node ID: 12a3qUuetnZPo3mAGDWaPQtWQe8v5AveuKqscgS3aH1N1VMyPjV, Piece ID: 3XFFIFZM3UOPDOSWPVHXJQBSUZZ273XG7IJ25H6Z77U5R5KBK5YQ) context deadline exceeded; piecestore: (Node ID: 12bGxp1vLoht7Be8UHdfPXW7z4g51HbLw9fDKRa9jgv1ebztzJC, Piece ID: 3H3LTDGAPSS6AIEQ4A3QPXUZTSQIUJO7WDZC2AUAUZNVFVMB5HXQ) context deadline exceeded; piecestore: (Node ID: 12LDFuZbL93YbimH8WpmMgmajNhpBnLZ1bMB7K7KW9WyNKYV6Zw, Piece ID: YHTBHCGJX5UREKEVI64QA6WJGLIUIJU7S52M5VOLBESFGFYEW3AA) context deadline exceeded; piecestore: (Node ID: 12MiSWqqubBGaQm54btriUNz2FqoLjjEgNakbxRQ2MLXWtDiqu6, Piece ID: IQXPA7FCB7WJFUI5XDCH3CYWJK3RB6YWBQFHNMPZNC7U6ZK3JTRQ) context deadline exceeded Duration: 6.321111308s core uplink eu1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.613565379s trace: https://jaeger.dev.storj.io/trace/3c6be275344fe875 FAIL (2026-01-03 13:00:30 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.613613881s core uplink ap1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 10 out of 60 is failed (threshold=3) [....X..X.....X...X................X.....X..X...X.X......X...] duration: 9.627210987s trace: https://jaeger.dev.storj.io/trace/53bc30794fd42896 FAIL (2026-01-03 13:00:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 22) ======LOGS====== 286]: Uploading piece... Jan 3 12:57:19.136747473 (693h28m19.007830508s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137034108 (693h28m19.008117143s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137189855 (693h28m19.008272891s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137413200 (693h28m19.008496245s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137560532 (693h28m19.008643567s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137902893 (693h28m19.008985938s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138193637 (693h28m19.009276692s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138313765 (693h28m19.0093968s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138696524 (693h28m19.009779569s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:20.027400153 (693h28m19.898483188s) [upload=607286]: Next returned nil segment Jan 3 12:57:20.027453063 (693h28m19.898536108s) [upload=607286]: Waiting for error group managing segments... Jan 3 12:57:20.410138690 (693h28m20.281221725s) [noise=true , address=185.167.204.205:4002 , piece_id=MOWRJZ3WLWZZXXB5ENPBOVZGVAD2V353VZ5CWCEO22PPZQGROVHQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:20.492391803 (693h28m20.363474848s) [noise=true , address=185.167.204.205:4022 , piece_id=CEGF62VNP7MZO7WADY2WL7WZX6UDOUA5NTWGGODTJGNKHBCCGJVA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.019412959 (693h28m20.890496004s) [noise=true , address=81.171.14.70:4030 , piece_id=GDSHOILKCEFT5BP2R6ZZDUQ5DF4CO5H3OD754ROW6GFRU3UU35TA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.054989251 (693h28m20.926072287s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.057129006 (693h28m20.928212052s) [noise=true , address=81.171.14.70:4028 , piece_id=P42SQIYXEJJKQEI2GFXRU5JIJ5G3XHLT2SM45DCE67E7S5TSPSYA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.060624582 (693h28m20.931707617s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.066618325 (693h28m20.93770136s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.076354184 (693h28m20.947437219s) [noise=true , address=81.171.14.70:4046 , piece_id=CXZXC744XXEYRZ252W2JM5N3VXFWA6UHUQYM6Z5WME7Q2B66YS5Q, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.084082891 (693h28m20.955165936s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.565324133 (693h28m21.436407168s) [noise=true , address=81.171.14.70:4032 , piece_id=5X2IPF6QMSEKVGMZFXBBW3DVXAK5SD3ZELAGWPRFN5C6SEVNUNWA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.584630255 (693h28m21.455713291s) [noise=true , address=81.171.14.70:4038 , piece_id=JI4QF46ZDL3TM4EFESCJSFFKIYRTF7CJZCC2HMUQZGO4CP2M4V5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.587294258 (693h28m21.458377303s) [noise=true , address=81.171.14.70:4034 , piece_id=GAFESRFTUPAOIHJ5LKEFK7AIZMLKL7JO5NCC26ZR27PQTTYH5UUA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.874175286 (693h28m21.745258321s) [noise=true , address=89.149.201.147:4016 , piece_id=MA3LBUYQ6OY7E4HMSROO5FVKJE7CEXF7SXUJIWWQX4EQCLI3OC7A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.284508037 (693h28m22.155591072s) [noise=true , address=89.149.201.147:4006 , piece_id=TQWVNQARGBZR2SRZ2LA3EVXSASEX36WXIEBGXWNWH2VQDSVC6DTA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.378387289 (693h28m22.249470334s) [noise=true , address=89.149.201.147:4015 , piece_id=SMHKGOSXQUWCO24KJLOH5RTLTZJ7MOADSCGPC6WRZUQFQHNNQ6WQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:23.055399573 (693h28m22.926482618s) [noise=true , address=81.171.14.70:4043 , piece_id=XKAAVOXN772VX5A6HSKBOU5NOYYF4U3HZ2BA5PYEOOQKCFGAIAFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:28.963038821 (693h28m28.834121857s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:32.765601718 (693h28m32.636684753s) [noise=true , address=185.167.204.205:4019 , piece_id=JKTSOUCJ2OF43PZKOKLRGUHSKRFWHQNMTIESGBK4MQNZO6VTWCLQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:36.941962859 (693h28m36.813045904s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:40.179862153 (693h28m40.050945188s) [noise=true , address=185.167.204.205:4010 , piece_id=CJBQASNMJAA5VBAFF65GU22ITWXU3RLHF6B7TFIDSUWCGSJM3OIA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:43.522593496 (693h28m43.393676531s) [noise=true , address=185.167.204.206:4041 , piece_id=QVVTZZDAAESA7IKRLWUMFMXOM4WIRUA6DP4FU2YKXSLK4WPTEC4A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:48.895462434 (693h28m48.766545479s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:58:18.486768101 (693h29m18.357851136s) [noise=true , address=185.167.204.205:4025 , piece_id=OFNEI5EOXKHZNLNDSJUQSMOBYUIGCFSAVLKRSJ5TKB3KX6ZXGN5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1cnhWXZCgZz567E5PAaPyfNKkmsSa4imaWPmfW2ir7U48gYEkp, address:185.167.204.205:4025): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487034448 (693h29m18.358117483s) [noise=true , address=185.167.204.206:4047 , piece_id=ZHLGMV427BQKUCYXOXRSMGVYTQH5TR7H62BKMOHY6QZHFQLCY2BQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487255509 (693h29m18.358338554s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12hK2MSVcFNaCVp7k4krUva7jXEJYhjnHPxPrMgq87jcDXJPkv2, address:185.167.204.206:4051): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487444569 (693h29m18.358527604s) [noise=true , address=185.167.204.205:4027 , piece_id=VIHPQE3Q7CRTAVPIYKTT4PIRU3RW4LPLUUWULRT66IX574KZ2GBA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487563535 (693h29m18.35864657s) [noise=true , address=185.167.204.205:4011 , piece_id=QBDCX7YUNQXSNX5BNO62KY4EZF7Y24KSMGDVR3XMBZF4JVM4C5FQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: read tcp 188.166.225.170:40296->185.167.204.205:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487627026 (693h29m18.358710072s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: read tcp 188.166.225.170:45010->185.167.204.205:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487692902 (693h29m18.358775937s) [seg_pos={0 0}, upload=607286]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil> Jan 3 12:58:18.668736929 (693h29m18.539819974s) [upload=607286]: Done uploading segments... Jan 3 12:58:18.668774270 (693h29m18.539857315s) [upload=607286]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22) Duration: 9.627242367s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........X.................X.....................] duration: 7.274958762s trace: https://jaeger.dev.storj.io/trace/640fd8ac8dea8303 FAIL (2026-01-03 13:00:10 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 7.275016151s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX.X..X.XX.X.....XXX.XXXX..X..X...X..XX..XX..X...XX..XX.X...] duration: 7.303467976s trace: https://jaeger.dev.storj.io/trace/6384b51b065162e6 FAIL (2026-01-03 13:00:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: 4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA) context deadline exceeded; piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: 3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA) context deadline exceeded ======LOGS====== Jan 3 12:57:32.139975894 (693h28m32.011058929s) [upload=607290]: Starting upload Jan 3 12:57:32.140035398 (693h28m32.011118433s) [upload=607290]: Done starting upload Jan 3 12:57:32.140163461 (693h28m32.011246496s) [upload=607290]: Uploading segments... Jan 3 12:57:32.140286756 (693h28m32.011369801s) [upload=607290]: Got next segment. Inline: false Jan 3 12:57:32.140334296 (693h28m32.011417331s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 3 12:57:32.140339225 (693h28m32.01142226s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 3 12:57:32.386239220 (693h28m32.257322265s) [seg_pos={0 0}, upload=607290]: Begin upload segment... Jan 3 12:57:32.386291079 (693h28m32.257374114s) [seg_pos={0 0}, upload=607290]: Done begin upload segment. Jan 3 12:57:32.386337226 (693h28m32.257420261s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386347246 (693h28m32.257430281s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386399996 (693h28m32.257483041s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386793715 (693h28m32.257876751s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387239204 (693h28m32.258322239s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387746029 (693h28m32.258829065s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.388247554 (693h28m32.259330619s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.390590996 (693h28m32.261674041s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.394182876 (693h28m32.265265912s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.397175685 (693h28m32.26825872s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.398532489 (693h28m32.269615534s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.400607720 (693h28m32.271690755s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.402897280 (693h28m32.273980315s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.403308622 (693h28m32.274391657s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.405982995 (693h28m32.27706603s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.407419360 (693h28m32.278502395s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:34.297376941 (693h28m34.168459976s) [upload=607290]: Next returned nil segment Jan 3 12:57:34.297397681 (693h28m34.168480716s) [upload=607290]: Waiting for error group managing segments... Jan 3 12:57:34.931863919 (693h28m34.802946954s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.117922626 (693h28m34.989005671s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.556098205 (693h28m35.42718125s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.595938106 (693h28m35.467021141s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.211248298 (693h28m36.082331333s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.430239493 (693h28m36.301322528s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.787016996 (693h28m36.658100031s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.437957986 (693h28m37.309041031s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.960941243 (693h28m37.832024278s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:38.634528192 (693h28m38.505611227s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.241920683 (693h28m40.113003718s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.826824296 (693h28m40.697907331s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612734057 (693h28m41.483817092s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612755909 (693h28m41.483838954s) [seg_pos={0 0}, upload=607290]: Segment reached optimal threshold of 13 pieces. Jan 3 12:57:41.613062503 (693h28m41.484145548s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: read tcp 188.166.225.170:48988->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:57:41.613069997 (693h28m41.484153032s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613316457 (693h28m41.484399492s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613324452 (693h28m41.484407487s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613457245 (693h28m41.48454029s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613494676 (693h28m41.484577731s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613539030 (693h28m41.484622075s) [seg_pos={0 0}, upload=607290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 3 12:57:41.613773797 (693h28m41.484856842s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:57:41.613782373 (693h28m41.484865428s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:57:41.908933460 (693h28m41.780016505s) [upload=607290]: Done uploading segments... Jan 3 12:57:41.908950523 (693h28m41.780033558s) [upload=607290]: Upload finished. err: <nil> Duration: 7.303501119s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [X...X.XX....XX.XXX.XXX.XXXX.XXXXX.XX.XXXXXX.XX.XXX.X.XX.X.XX] duration: 30.001064671s trace: https://jaeger.dev.storj.io/trace/9f15a1c78f3fcc2 FAIL (2026-01-03 13:00:06 UTC) orbiter warning: context deadline exceeded Duration: 30.001083297s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................X.X........X.............X................] duration: 21.383679819s trace: https://jaeger.dev.storj.io/trace/165637e999409e3 FAIL (2026-01-03 12:59:53 UTC) orbiter warning: context deadline exceeded Duration: 21.383710597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XX.X..X.XX.X.....XXX.XXXX..X..X...X..XX..XX..X...XX..XX.X..] duration: 16.540923384s trace: https://jaeger.dev.storj.io/trace/6384b51b065162e6 FAIL (2026-01-03 12:59:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: 4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA) context deadline exceeded; piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: 3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA) context deadline exceeded ======LOGS====== Jan 3 12:57:32.139975894 (693h28m32.011058929s) [upload=607290]: Starting upload Jan 3 12:57:32.140035398 (693h28m32.011118433s) [upload=607290]: Done starting upload Jan 3 12:57:32.140163461 (693h28m32.011246496s) [upload=607290]: Uploading segments... Jan 3 12:57:32.140286756 (693h28m32.011369801s) [upload=607290]: Got next segment. Inline: false Jan 3 12:57:32.140334296 (693h28m32.011417331s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 3 12:57:32.140339225 (693h28m32.01142226s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 3 12:57:32.386239220 (693h28m32.257322265s) [seg_pos={0 0}, upload=607290]: Begin upload segment... Jan 3 12:57:32.386291079 (693h28m32.257374114s) [seg_pos={0 0}, upload=607290]: Done begin upload segment. Jan 3 12:57:32.386337226 (693h28m32.257420261s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386347246 (693h28m32.257430281s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386399996 (693h28m32.257483041s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386793715 (693h28m32.257876751s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387239204 (693h28m32.258322239s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387746029 (693h28m32.258829065s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.388247554 (693h28m32.259330619s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.390590996 (693h28m32.261674041s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.394182876 (693h28m32.265265912s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.397175685 (693h28m32.26825872s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.398532489 (693h28m32.269615534s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.400607720 (693h28m32.271690755s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.402897280 (693h28m32.273980315s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.403308622 (693h28m32.274391657s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.405982995 (693h28m32.27706603s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.407419360 (693h28m32.278502395s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:34.297376941 (693h28m34.168459976s) [upload=607290]: Next returned nil segment Jan 3 12:57:34.297397681 (693h28m34.168480716s) [upload=607290]: Waiting for error group managing segments... Jan 3 12:57:34.931863919 (693h28m34.802946954s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.117922626 (693h28m34.989005671s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.556098205 (693h28m35.42718125s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.595938106 (693h28m35.467021141s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.211248298 (693h28m36.082331333s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.430239493 (693h28m36.301322528s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.787016996 (693h28m36.658100031s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.437957986 (693h28m37.309041031s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.960941243 (693h28m37.832024278s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:38.634528192 (693h28m38.505611227s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.241920683 (693h28m40.113003718s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.826824296 (693h28m40.697907331s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612734057 (693h28m41.483817092s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612755909 (693h28m41.483838954s) [seg_pos={0 0}, upload=607290]: Segment reached optimal threshold of 13 pieces. Jan 3 12:57:41.613062503 (693h28m41.484145548s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: read tcp 188.166.225.170:48988->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:57:41.613069997 (693h28m41.484153032s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613316457 (693h28m41.484399492s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613324452 (693h28m41.484407487s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613457245 (693h28m41.48454029s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613494676 (693h28m41.484577731s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613539030 (693h28m41.484622075s) [seg_pos={0 0}, upload=607290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 3 12:57:41.613773797 (693h28m41.484856842s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:57:41.613782373 (693h28m41.484865428s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:57:41.908933460 (693h28m41.780016505s) [upload=607290]: Done uploading segments... Jan 3 12:57:41.908950523 (693h28m41.780033558s) [upload=607290]: Upload finished. err: <nil> Duration: 16.541013855s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXX.X.XXXXXXX.XXXX.XXX.XX.XX....X.XXXXXXX.] duration: 4.289177372s trace: https://jaeger.dev.storj.io/trace/4c5a20512753713c FAIL (2026-01-03 12:59:48 UTC) orbiter warning: context deadline exceeded Duration: 4.289311797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.....................X.......] duration: 8.843708017s trace: https://jaeger.dev.storj.io/trace/2a592caa2cd77abc FAIL (2026-01-03 12:59:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.843723666s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.034181667s trace: https://jaeger.dev.storj.io/trace/5a0aed6ab18c8274 FAIL (2026-01-03 12:59:41 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 1F3yVaqH52tef8gWWe4JAWh47yRjDDEqYyou7abzPnDkDEeson, Piece ID: FEJXLLJSEBSH4XFEJX54FSB3U3237HMWQWZJAIGHDRFQHPNB5KLA) context deadline exceeded; piecestore: (Node ID: 1yyv7LXhZGJFvkTV4vafuZ6KrQkBWB7zH7q2emQ3dNat8iuWsK, Piece ID: OWDX5T5OCZF6L7DTGORIV24KCTBVMI5NUWJC5JZVGYOR36KS3DPQ) context deadline exceeded; piecestore: (Node ID: 12q1oXSNGG6uCuHBtvGTxrbWLnchY11hmhfbiRzcm1zCwGvz7Zh, Piece ID: T4E3TYY6CEPW54CARC2EIJS6YF4CDABBI7LIJQX5Q6AVFO7JEVNQ) context deadline exceeded; piecestore: (Node ID: 1UtnhDdcRHT4L5h1MxGCRWfY24bCETpRgMz62K8s4qAMkwS3fG, Piece ID: 64IAPWLKLFHF7MI3IAHFZHBNOTH64G4OPBC2NGPL5DZCWJHBXOFQ) context deadline exceeded; piecestore: (Node ID: 1x29ZBapLNUJt9E9WjyjUYav5LK4d3h5MsmZAVH2L2daqiANw1, Piece ID: QPZFZ3N4E2VSRMUB7XU6LZUH4WJFVEFYQMLTZ6O7ACG5RMBHNDJA) context deadline exceeded; piecestore: (Node ID: 1EQTE1zDuMWpoXmTB2jkH9cv2Pe2yAGWCc6A4iNznkSKzkN8pM, Piece ID: SAYUQDKORTF7IF7XHIUXCI454RUHCCBQPCIBKMX6T2E6LR2DRRUA) context deadline exceeded; piecestore: (Node ID: 1o2qgqfNpCnXxd66kmxTTqeQzWzuegmzrhqkP9bZ9CXmjLzva5, Piece ID: Q5AR3ZKTR6TX2ZQAH34DDVLT7IP5Y5N4HKSHNJPJQMS6QNTRGXIA) context deadline exceeded; piecestore: (Node ID: 12a3qUuetnZPo3mAGDWaPQtWQe8v5AveuKqscgS3aH1N1VMyPjV, Piece ID: 3XFFIFZM3UOPDOSWPVHXJQBSUZZ273XG7IJ25H6Z77U5R5KBK5YQ) context deadline exceeded; piecestore: (Node ID: 12bGxp1vLoht7Be8UHdfPXW7z4g51HbLw9fDKRa9jgv1ebztzJC, Piece ID: 3H3LTDGAPSS6AIEQ4A3QPXUZTSQIUJO7WDZC2AUAUZNVFVMB5HXQ) context deadline exceeded; piecestore: (Node ID: 12LDFuZbL93YbimH8WpmMgmajNhpBnLZ1bMB7K7KW9WyNKYV6Zw, Piece ID: YHTBHCGJX5UREKEVI64QA6WJGLIUIJU7S52M5VOLBESFGFYEW3AA) context deadline exceeded; piecestore: (Node ID: 12MiSWqqubBGaQm54btriUNz2FqoLjjEgNakbxRQ2MLXWtDiqu6, Piece ID: IQXPA7FCB7WJFUI5XDCH3CYWJK3RB6YWBQFHNMPZNC7U6ZK3JTRQ) context deadline exceeded Duration: 6.034220089s core uplink eu1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.856573171s trace: https://jaeger.dev.storj.io/trace/3c6be275344fe875 FAIL (2026-01-03 12:59:34 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 3.856599351s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXX.X...X...XX.X.X.XXX.XXXXX..XX...X.X.X..XX.XXX] duration: 59.117111393s trace: https://jaeger.dev.storj.io/trace/3c0af02f62dac641 FAIL (2026-01-03 12:59:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== PA, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.371768077 (693h29m35.242851112s) [noise=true , address=145.79.227.17:4981 , piece_id=2F4I7GV4YWVZ55YOFR7SJ3ELBMBK2NHBID3ILT3NPTCRVJQH7S5Q, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.372186734 (693h29m35.24326978s) [noise=true , address=77.90.31.18:4828 , piece_id=BEPTU5OPE5HU3DUXUYW3WJUUCXCJUTTRSO24KWBBQI6AHAPAVWLQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.372467749 (693h29m35.243550795s) [noise=true , address=193.149.164.143:4702 , piece_id=EXI6MYYY7RQKNLWVBVGSGSDGJHHLB2JTNEXO54ZZPEQVY24G5SIQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.372758984 (693h29m35.243842029s) [noise=true , address=77.90.31.10:4548 , piece_id=MMRKLPAESMH4PZ4ALOWPAEGJUPMCYQPRCSLMQQBB6MLDXPREXFRA, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.373052303 (693h29m35.244135338s) [noise=true , address=45.91.116.6:4994 , piece_id=EQFJMH2S7LGXZS2E4EK6MYMGUB4D24EBJ2ZCKYMHZAKE43AMQ3HQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.373415704 (693h29m35.244498739s) [noise=true , address=45.91.116.5:4303 , piece_id=R2PFBO3TH3YK3TEJKDL6B2AI2UQF44XG6T7OOPBEEBQYNCTZY3HQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.373826677 (693h29m35.244909712s) [noise=true , address=145.79.227.34:4918 , piece_id=S3F6IXAVMLFRXV3PBEDHTMZ7GSXVHELF2W2GEZZNMDXIPTEAKGKQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.375598351 (693h29m35.246681386s) [noise=true , address=45.91.116.8:4078 , piece_id=N5LQQ577PDOSG6SRMF7BWIWB7TU3DDPJQCW3U64VZTKINXMMOZOQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.376713935 (693h29m35.24779697s) [noise=true , address=45.91.116.14:4221 , piece_id=X3ZIT67NO45GDYNSWJJACUSFKID5U2PGDWH36ZTYXAOWIQLPENKA, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.377885506 (693h29m35.248968541s) [noise=true , address=77.90.31.13:4637 , piece_id=DCDWWVWNFWSO4EKXBP5M2UJVUKEZCVXG5PATD3XCSQEPVUZVYPWA, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.379123804 (693h29m35.250206839s) [noise=true , address=145.79.227.43:4186 , piece_id=MWVITXOSOYICOSBFX753GYTU2D57FMOHCLC5DWZOJKK4XO6J6VXQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.380398151 (693h29m35.251481196s) [noise=true , address=145.79.227.35:4515 , piece_id=YFQX6OIFD7IBI6ANTUD2QZ7JXXHZYWCBG4KQVHAKNBCIHPTCPKLQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.381748251 (693h29m35.252831286s) [noise=true , address=145.79.227.44:4205 , piece_id=7SKZGNJ26VCXNWTWNVJOMT6UYYDE2SCDSO65OY655THTOS27YH5Q, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.382890826 (693h29m35.253973871s) [noise=true , address=145.79.227.40:4078 , piece_id=C7MTD5IIDARBAXQDQZM6ILJ2QBUPIYRPLGQVITZSSCYCWAU5RYPQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:35.383066541 (693h29m35.254149576s) [noise=true , address=64.49.13.169:4706 , piece_id=OGS2NCYIJUAFE7TYVDOCXSUGLJATNT454GCBLUSG7JYLCQILCPAQ, seg_pos={0 0}, upload=607303]: Uploading piece... Jan 3 12:58:36.791937624 (693h29m36.663020659s) [upload=607303]: Next returned nil segment Jan 3 12:58:36.791954476 (693h29m36.663037511s) [upload=607303]: Waiting for error group managing segments... Jan 3 12:58:37.224088932 (693h29m37.095171977s) [noise=true , address=145.79.227.17:4981 , piece_id=2F4I7GV4YWVZ55YOFR7SJ3ELBMBK2NHBID3ILT3NPTCRVJQH7S5Q, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:38.213417086 (693h29m38.084500121s) [noise=true , address=145.79.227.44:4205 , piece_id=7SKZGNJ26VCXNWTWNVJOMT6UYYDE2SCDSO65OY655THTOS27YH5Q, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:38.214807514 (693h29m38.085890549s) [noise=true , address=145.79.227.34:4918 , piece_id=S3F6IXAVMLFRXV3PBEDHTMZ7GSXVHELF2W2GEZZNMDXIPTEAKGKQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:38.224176704 (693h29m38.095259749s) [noise=true , address=145.79.227.35:4515 , piece_id=YFQX6OIFD7IBI6ANTUD2QZ7JXXHZYWCBG4KQVHAKNBCIHPTCPKLQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:38.227653034 (693h29m38.098736069s) [noise=true , address=145.79.227.30:4782 , piece_id=WCABXEEJIJ7RNWBEK67J6XO2NAKSF5GPAG5JC5YB7HXYFIGMRL4Q, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:38.260250885 (693h29m38.131333921s) [noise=true , address=145.79.227.40:4078 , piece_id=C7MTD5IIDARBAXQDQZM6ILJ2QBUPIYRPLGQVITZSSCYCWAU5RYPQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:38.322391801 (693h29m38.193474836s) [noise=true , address=145.79.227.43:4186 , piece_id=MWVITXOSOYICOSBFX753GYTU2D57FMOHCLC5DWZOJKK4XO6J6VXQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:39.351377259 (693h29m39.222460304s) [noise=true , address=77.90.31.12:4599 , piece_id=6UG26REL3HJLBZNHCMNUHS62COIJ4E56GKJNBL6ELPPFFFLC7LPA, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:39.722734124 (693h29m39.593817159s) [noise=true , address=45.91.116.13:4182 , piece_id=SOCHJZRV3HZCJQGX34MZ2PABO37AYKZ2SHROU4EF6VHFWTVCD2BA, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:39.778855047 (693h29m39.649938092s) [noise=true , address=45.91.116.6:4994 , piece_id=EQFJMH2S7LGXZS2E4EK6MYMGUB4D24EBJ2ZCKYMHZAKE43AMQ3HQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:40.720464838 (693h29m40.591547883s) [noise=true , address=193.149.164.143:4702 , piece_id=EXI6MYYY7RQKNLWVBVGSGSDGJHHLB2JTNEXO54ZZPEQVY24G5SIQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:41.720267568 (693h29m41.591350603s) [noise=true , address=77.90.31.13:4637 , piece_id=DCDWWVWNFWSO4EKXBP5M2UJVUKEZCVXG5PATD3XCSQEPVUZVYPWA, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:42.186782197 (693h29m42.057865243s) [noise=true , address=45.91.116.8:4078 , piece_id=N5LQQ577PDOSG6SRMF7BWIWB7TU3DDPJQCW3U64VZTKINXMMOZOQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:42.302417085 (693h29m42.173500131s) [noise=true , address=77.90.31.10:4548 , piece_id=MMRKLPAESMH4PZ4ALOWPAEGJUPMCYQPRCSLMQQBB6MLDXPREXFRA, seg_pos={0 0}, upload=607303]: Done uploading piece. err: <nil> Jan 3 12:58:42.302442845 (693h29m42.17352589s) [seg_pos={0 0}, upload=607303]: Segment reached optimal threshold of 14 pieces. Jan 3 12:58:42.302649178 (693h29m42.173732213s) [noise=true , address=64.49.13.169:4706 , piece_id=OGS2NCYIJUAFE7TYVDOCXSUGLJATNT454GCBLUSG7JYLCQILCPAQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51336->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51336->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51336->64.49.13.169:4706: use of closed network connection Jan 3 12:58:42.302657824 (693h29m42.173740859s) [noise=true , address=64.49.13.169:4706 , piece_id=OGS2NCYIJUAFE7TYVDOCXSUGLJATNT454GCBLUSG7JYLCQILCPAQ, seg_pos={0 0}, upload=607303]: Long tail cancellation due to optimal threshold Jan 3 12:58:42.302818029 (693h29m42.173901074s) [noise=true , address=45.91.116.5:4303 , piece_id=R2PFBO3TH3YK3TEJKDL6B2AI2UQF44XG6T7OOPBEEBQYNCTZY3HQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY): protocol: expected piece hash; context canceled; EOF Jan 3 12:58:42.302827267 (693h29m42.173910332s) [noise=true , address=45.91.116.5:4303 , piece_id=R2PFBO3TH3YK3TEJKDL6B2AI2UQF44XG6T7OOPBEEBQYNCTZY3HQ, seg_pos={0 0}, upload=607303]: Long tail cancellation due to optimal threshold Jan 3 12:58:42.303020765 (693h29m42.17410382s) [noise=true , address=45.91.116.14:4221 , piece_id=X3ZIT67NO45GDYNSWJJACUSFKID5U2PGDWH36ZTYXAOWIQLPENKA, seg_pos={0 0}, upload=607303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jZ4TRrJvrBgeWJfnpN7tskdv5rPR6LFuSYo814igtdwMDg8ZX): protocol: expected piece hash; context canceled; EOF Jan 3 12:58:42.303029121 (693h29m42.174112166s) [noise=true , address=45.91.116.14:4221 , piece_id=X3ZIT67NO45GDYNSWJJACUSFKID5U2PGDWH36ZTYXAOWIQLPENKA, seg_pos={0 0}, upload=607303]: Long tail cancellation due to optimal threshold Jan 3 12:58:42.303047896 (693h29m42.174130932s) [noise=true , address=77.90.31.18:4828 , piece_id=BEPTU5OPE5HU3DUXUYW3WJUUCXCJUTTRSO24KWBBQI6AHAPAVWLQ, seg_pos={0 0}, upload=607303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9): protocol: read tcp 188.166.225.170:43476->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:42.303056643 (693h29m42.174139678s) [noise=true , address=77.90.31.18:4828 , piece_id=BEPTU5OPE5HU3DUXUYW3WJUUCXCJUTTRSO24KWBBQI6AHAPAVWLQ, seg_pos={0 0}, upload=607303]: Long tail cancellation due to optimal threshold Jan 3 12:58:42.303072423 (693h29m42.174155459s) [seg_pos={0 0}, upload=607303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 3 12:58:42.303155762 (693h29m42.174238807s) [upload=607303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:58:42.303163648 (693h29m42.174246683s) [upload=607303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:58:42.584657378 (693h29m42.455740433s) [upload=607303]: Done uploading segments... Jan 3 12:58:42.584689478 (693h29m42.455772523s) [upload=607303]: Upload finished. err: <nil> Duration: 59.117149045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 10 out of 60 is failed (threshold=3) [.....X..X.....X...X................X.....X..X...X.X......X..] duration: 11.770586973s trace: https://jaeger.dev.storj.io/trace/53bc30794fd42896 FAIL (2026-01-03 12:59:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 22) ======LOGS====== 286]: Uploading piece... Jan 3 12:57:19.136747473 (693h28m19.007830508s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137034108 (693h28m19.008117143s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137189855 (693h28m19.008272891s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137413200 (693h28m19.008496245s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137560532 (693h28m19.008643567s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137902893 (693h28m19.008985938s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138193637 (693h28m19.009276692s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138313765 (693h28m19.0093968s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138696524 (693h28m19.009779569s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:20.027400153 (693h28m19.898483188s) [upload=607286]: Next returned nil segment Jan 3 12:57:20.027453063 (693h28m19.898536108s) [upload=607286]: Waiting for error group managing segments... Jan 3 12:57:20.410138690 (693h28m20.281221725s) [noise=true , address=185.167.204.205:4002 , piece_id=MOWRJZ3WLWZZXXB5ENPBOVZGVAD2V353VZ5CWCEO22PPZQGROVHQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:20.492391803 (693h28m20.363474848s) [noise=true , address=185.167.204.205:4022 , piece_id=CEGF62VNP7MZO7WADY2WL7WZX6UDOUA5NTWGGODTJGNKHBCCGJVA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.019412959 (693h28m20.890496004s) [noise=true , address=81.171.14.70:4030 , piece_id=GDSHOILKCEFT5BP2R6ZZDUQ5DF4CO5H3OD754ROW6GFRU3UU35TA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.054989251 (693h28m20.926072287s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.057129006 (693h28m20.928212052s) [noise=true , address=81.171.14.70:4028 , piece_id=P42SQIYXEJJKQEI2GFXRU5JIJ5G3XHLT2SM45DCE67E7S5TSPSYA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.060624582 (693h28m20.931707617s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.066618325 (693h28m20.93770136s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.076354184 (693h28m20.947437219s) [noise=true , address=81.171.14.70:4046 , piece_id=CXZXC744XXEYRZ252W2JM5N3VXFWA6UHUQYM6Z5WME7Q2B66YS5Q, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.084082891 (693h28m20.955165936s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.565324133 (693h28m21.436407168s) [noise=true , address=81.171.14.70:4032 , piece_id=5X2IPF6QMSEKVGMZFXBBW3DVXAK5SD3ZELAGWPRFN5C6SEVNUNWA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.584630255 (693h28m21.455713291s) [noise=true , address=81.171.14.70:4038 , piece_id=JI4QF46ZDL3TM4EFESCJSFFKIYRTF7CJZCC2HMUQZGO4CP2M4V5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.587294258 (693h28m21.458377303s) [noise=true , address=81.171.14.70:4034 , piece_id=GAFESRFTUPAOIHJ5LKEFK7AIZMLKL7JO5NCC26ZR27PQTTYH5UUA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.874175286 (693h28m21.745258321s) [noise=true , address=89.149.201.147:4016 , piece_id=MA3LBUYQ6OY7E4HMSROO5FVKJE7CEXF7SXUJIWWQX4EQCLI3OC7A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.284508037 (693h28m22.155591072s) [noise=true , address=89.149.201.147:4006 , piece_id=TQWVNQARGBZR2SRZ2LA3EVXSASEX36WXIEBGXWNWH2VQDSVC6DTA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.378387289 (693h28m22.249470334s) [noise=true , address=89.149.201.147:4015 , piece_id=SMHKGOSXQUWCO24KJLOH5RTLTZJ7MOADSCGPC6WRZUQFQHNNQ6WQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:23.055399573 (693h28m22.926482618s) [noise=true , address=81.171.14.70:4043 , piece_id=XKAAVOXN772VX5A6HSKBOU5NOYYF4U3HZ2BA5PYEOOQKCFGAIAFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:28.963038821 (693h28m28.834121857s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:32.765601718 (693h28m32.636684753s) [noise=true , address=185.167.204.205:4019 , piece_id=JKTSOUCJ2OF43PZKOKLRGUHSKRFWHQNMTIESGBK4MQNZO6VTWCLQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:36.941962859 (693h28m36.813045904s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:40.179862153 (693h28m40.050945188s) [noise=true , address=185.167.204.205:4010 , piece_id=CJBQASNMJAA5VBAFF65GU22ITWXU3RLHF6B7TFIDSUWCGSJM3OIA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:43.522593496 (693h28m43.393676531s) [noise=true , address=185.167.204.206:4041 , piece_id=QVVTZZDAAESA7IKRLWUMFMXOM4WIRUA6DP4FU2YKXSLK4WPTEC4A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:48.895462434 (693h28m48.766545479s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:58:18.486768101 (693h29m18.357851136s) [noise=true , address=185.167.204.205:4025 , piece_id=OFNEI5EOXKHZNLNDSJUQSMOBYUIGCFSAVLKRSJ5TKB3KX6ZXGN5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1cnhWXZCgZz567E5PAaPyfNKkmsSa4imaWPmfW2ir7U48gYEkp, address:185.167.204.205:4025): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487034448 (693h29m18.358117483s) [noise=true , address=185.167.204.206:4047 , piece_id=ZHLGMV427BQKUCYXOXRSMGVYTQH5TR7H62BKMOHY6QZHFQLCY2BQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487255509 (693h29m18.358338554s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12hK2MSVcFNaCVp7k4krUva7jXEJYhjnHPxPrMgq87jcDXJPkv2, address:185.167.204.206:4051): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487444569 (693h29m18.358527604s) [noise=true , address=185.167.204.205:4027 , piece_id=VIHPQE3Q7CRTAVPIYKTT4PIRU3RW4LPLUUWULRT66IX574KZ2GBA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487563535 (693h29m18.35864657s) [noise=true , address=185.167.204.205:4011 , piece_id=QBDCX7YUNQXSNX5BNO62KY4EZF7Y24KSMGDVR3XMBZF4JVM4C5FQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: read tcp 188.166.225.170:40296->185.167.204.205:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487627026 (693h29m18.358710072s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: read tcp 188.166.225.170:45010->185.167.204.205:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487692902 (693h29m18.358775937s) [seg_pos={0 0}, upload=607286]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil> Jan 3 12:58:18.668736929 (693h29m18.539819974s) [upload=607286]: Done uploading segments... Jan 3 12:58:18.668774270 (693h29m18.539857315s) [upload=607286]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22) Duration: 11.770612s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.X...X.XX....XX.XXX.XXX.XXXX.XXXXX.XX.XXXXXX.XX.XXX.X.XX.X.X] duration: 30.000739317s trace: https://jaeger.dev.storj.io/trace/5984eafe09d93fb1 FAIL (2026-01-03 12:59:17 UTC) orbiter warning: context deadline exceeded Duration: 30.000775586s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........X.................X....................] duration: 6.315152737s trace: https://jaeger.dev.storj.io/trace/640fd8ac8dea8303 FAIL (2026-01-03 12:59:11 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.315183524s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXX.X.XXXXXXX.XXXX.XXX.XX.XX....X.XXXXXXX] duration: 5.000991858s trace: https://jaeger.dev.storj.io/trace/4c5a20512753713c FAIL (2026-01-03 12:59:07 UTC) orbiter warning: context deadline exceeded Duration: 5.001015734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.851261462s trace: https://jaeger.dev.storj.io/trace/5a0aed6ab18c8274 FAIL (2026-01-03 12:59:04 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 1F3yVaqH52tef8gWWe4JAWh47yRjDDEqYyou7abzPnDkDEeson, Piece ID: FEJXLLJSEBSH4XFEJX54FSB3U3237HMWQWZJAIGHDRFQHPNB5KLA) context deadline exceeded; piecestore: (Node ID: 1yyv7LXhZGJFvkTV4vafuZ6KrQkBWB7zH7q2emQ3dNat8iuWsK, Piece ID: OWDX5T5OCZF6L7DTGORIV24KCTBVMI5NUWJC5JZVGYOR36KS3DPQ) context deadline exceeded; piecestore: (Node ID: 12q1oXSNGG6uCuHBtvGTxrbWLnchY11hmhfbiRzcm1zCwGvz7Zh, Piece ID: T4E3TYY6CEPW54CARC2EIJS6YF4CDABBI7LIJQX5Q6AVFO7JEVNQ) context deadline exceeded; piecestore: (Node ID: 1UtnhDdcRHT4L5h1MxGCRWfY24bCETpRgMz62K8s4qAMkwS3fG, Piece ID: 64IAPWLKLFHF7MI3IAHFZHBNOTH64G4OPBC2NGPL5DZCWJHBXOFQ) context deadline exceeded; piecestore: (Node ID: 1x29ZBapLNUJt9E9WjyjUYav5LK4d3h5MsmZAVH2L2daqiANw1, Piece ID: QPZFZ3N4E2VSRMUB7XU6LZUH4WJFVEFYQMLTZ6O7ACG5RMBHNDJA) context deadline exceeded; piecestore: (Node ID: 1EQTE1zDuMWpoXmTB2jkH9cv2Pe2yAGWCc6A4iNznkSKzkN8pM, Piece ID: SAYUQDKORTF7IF7XHIUXCI454RUHCCBQPCIBKMX6T2E6LR2DRRUA) context deadline exceeded; piecestore: (Node ID: 1o2qgqfNpCnXxd66kmxTTqeQzWzuegmzrhqkP9bZ9CXmjLzva5, Piece ID: Q5AR3ZKTR6TX2ZQAH34DDVLT7IP5Y5N4HKSHNJPJQMS6QNTRGXIA) context deadline exceeded; piecestore: (Node ID: 12a3qUuetnZPo3mAGDWaPQtWQe8v5AveuKqscgS3aH1N1VMyPjV, Piece ID: 3XFFIFZM3UOPDOSWPVHXJQBSUZZ273XG7IJ25H6Z77U5R5KBK5YQ) context deadline exceeded; piecestore: (Node ID: 12bGxp1vLoht7Be8UHdfPXW7z4g51HbLw9fDKRa9jgv1ebztzJC, Piece ID: 3H3LTDGAPSS6AIEQ4A3QPXUZTSQIUJO7WDZC2AUAUZNVFVMB5HXQ) context deadline exceeded; piecestore: (Node ID: 12LDFuZbL93YbimH8WpmMgmajNhpBnLZ1bMB7K7KW9WyNKYV6Zw, Piece ID: YHTBHCGJX5UREKEVI64QA6WJGLIUIJU7S52M5VOLBESFGFYEW3AA) context deadline exceeded; piecestore: (Node ID: 12MiSWqqubBGaQm54btriUNz2FqoLjjEgNakbxRQ2MLXWtDiqu6, Piece ID: IQXPA7FCB7WJFUI5XDCH3CYWJK3RB6YWBQFHNMPZNC7U6ZK3JTRQ) context deadline exceeded Duration: 5.851288643s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................X.X........X.............X...............] duration: 26.208214784s trace: https://jaeger.dev.storj.io/trace/165637e999409e3 FAIL (2026-01-03 12:59:04 UTC) orbiter warning: context deadline exceeded Duration: 26.20826087s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.....................X......] duration: 21.370708414s trace: https://jaeger.dev.storj.io/trace/2a592caa2cd77abc FAIL (2026-01-03 12:59:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.370740716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........X.................X...................] duration: 6.937605377s trace: https://jaeger.dev.storj.io/trace/640fd8ac8dea8303 FAIL (2026-01-03 12:58:51 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.937634242s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.750545343s trace: https://jaeger.dev.storj.io/trace/3c6be275344fe875 FAIL (2026-01-03 12:58:46 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.750568306s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XX.X..X.XX.X.....XXX.XXXX..X..X...X..XX..XX..X...XX..XX.X.] duration: 5.731090591s trace: https://jaeger.dev.storj.io/trace/6384b51b065162e6 FAIL (2026-01-03 12:58:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: 4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA) context deadline exceeded; piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: 3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA) context deadline exceeded ======LOGS====== Jan 3 12:57:32.139975894 (693h28m32.011058929s) [upload=607290]: Starting upload Jan 3 12:57:32.140035398 (693h28m32.011118433s) [upload=607290]: Done starting upload Jan 3 12:57:32.140163461 (693h28m32.011246496s) [upload=607290]: Uploading segments... Jan 3 12:57:32.140286756 (693h28m32.011369801s) [upload=607290]: Got next segment. Inline: false Jan 3 12:57:32.140334296 (693h28m32.011417331s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 3 12:57:32.140339225 (693h28m32.01142226s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 3 12:57:32.386239220 (693h28m32.257322265s) [seg_pos={0 0}, upload=607290]: Begin upload segment... Jan 3 12:57:32.386291079 (693h28m32.257374114s) [seg_pos={0 0}, upload=607290]: Done begin upload segment. Jan 3 12:57:32.386337226 (693h28m32.257420261s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386347246 (693h28m32.257430281s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386399996 (693h28m32.257483041s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386793715 (693h28m32.257876751s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387239204 (693h28m32.258322239s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387746029 (693h28m32.258829065s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.388247554 (693h28m32.259330619s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.390590996 (693h28m32.261674041s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.394182876 (693h28m32.265265912s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.397175685 (693h28m32.26825872s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.398532489 (693h28m32.269615534s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.400607720 (693h28m32.271690755s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.402897280 (693h28m32.273980315s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.403308622 (693h28m32.274391657s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.405982995 (693h28m32.27706603s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.407419360 (693h28m32.278502395s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:34.297376941 (693h28m34.168459976s) [upload=607290]: Next returned nil segment Jan 3 12:57:34.297397681 (693h28m34.168480716s) [upload=607290]: Waiting for error group managing segments... Jan 3 12:57:34.931863919 (693h28m34.802946954s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.117922626 (693h28m34.989005671s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.556098205 (693h28m35.42718125s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.595938106 (693h28m35.467021141s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.211248298 (693h28m36.082331333s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.430239493 (693h28m36.301322528s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.787016996 (693h28m36.658100031s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.437957986 (693h28m37.309041031s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.960941243 (693h28m37.832024278s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:38.634528192 (693h28m38.505611227s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.241920683 (693h28m40.113003718s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.826824296 (693h28m40.697907331s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612734057 (693h28m41.483817092s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612755909 (693h28m41.483838954s) [seg_pos={0 0}, upload=607290]: Segment reached optimal threshold of 13 pieces. Jan 3 12:57:41.613062503 (693h28m41.484145548s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: read tcp 188.166.225.170:48988->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:57:41.613069997 (693h28m41.484153032s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613316457 (693h28m41.484399492s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613324452 (693h28m41.484407487s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613457245 (693h28m41.48454029s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613494676 (693h28m41.484577731s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613539030 (693h28m41.484622075s) [seg_pos={0 0}, upload=607290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 3 12:57:41.613773797 (693h28m41.484856842s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:57:41.613782373 (693h28m41.484865428s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:57:41.908933460 (693h28m41.780016505s) [upload=607290]: Done uploading segments... Jan 3 12:57:41.908950523 (693h28m41.780033558s) [upload=607290]: Upload finished. err: <nil> Duration: 5.731264432s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 10 out of 60 is failed (threshold=3) [......X..X.....X...X................X.....X..X...X.X......X.] duration: 19.4160007s trace: https://jaeger.dev.storj.io/trace/53bc30794fd42896 FAIL (2026-01-03 12:58:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 22) ======LOGS====== 286]: Uploading piece... Jan 3 12:57:19.136747473 (693h28m19.007830508s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137034108 (693h28m19.008117143s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137189855 (693h28m19.008272891s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137413200 (693h28m19.008496245s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137560532 (693h28m19.008643567s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137902893 (693h28m19.008985938s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138193637 (693h28m19.009276692s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138313765 (693h28m19.0093968s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138696524 (693h28m19.009779569s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:20.027400153 (693h28m19.898483188s) [upload=607286]: Next returned nil segment Jan 3 12:57:20.027453063 (693h28m19.898536108s) [upload=607286]: Waiting for error group managing segments... Jan 3 12:57:20.410138690 (693h28m20.281221725s) [noise=true , address=185.167.204.205:4002 , piece_id=MOWRJZ3WLWZZXXB5ENPBOVZGVAD2V353VZ5CWCEO22PPZQGROVHQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:20.492391803 (693h28m20.363474848s) [noise=true , address=185.167.204.205:4022 , piece_id=CEGF62VNP7MZO7WADY2WL7WZX6UDOUA5NTWGGODTJGNKHBCCGJVA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.019412959 (693h28m20.890496004s) [noise=true , address=81.171.14.70:4030 , piece_id=GDSHOILKCEFT5BP2R6ZZDUQ5DF4CO5H3OD754ROW6GFRU3UU35TA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.054989251 (693h28m20.926072287s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.057129006 (693h28m20.928212052s) [noise=true , address=81.171.14.70:4028 , piece_id=P42SQIYXEJJKQEI2GFXRU5JIJ5G3XHLT2SM45DCE67E7S5TSPSYA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.060624582 (693h28m20.931707617s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.066618325 (693h28m20.93770136s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.076354184 (693h28m20.947437219s) [noise=true , address=81.171.14.70:4046 , piece_id=CXZXC744XXEYRZ252W2JM5N3VXFWA6UHUQYM6Z5WME7Q2B66YS5Q, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.084082891 (693h28m20.955165936s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.565324133 (693h28m21.436407168s) [noise=true , address=81.171.14.70:4032 , piece_id=5X2IPF6QMSEKVGMZFXBBW3DVXAK5SD3ZELAGWPRFN5C6SEVNUNWA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.584630255 (693h28m21.455713291s) [noise=true , address=81.171.14.70:4038 , piece_id=JI4QF46ZDL3TM4EFESCJSFFKIYRTF7CJZCC2HMUQZGO4CP2M4V5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.587294258 (693h28m21.458377303s) [noise=true , address=81.171.14.70:4034 , piece_id=GAFESRFTUPAOIHJ5LKEFK7AIZMLKL7JO5NCC26ZR27PQTTYH5UUA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.874175286 (693h28m21.745258321s) [noise=true , address=89.149.201.147:4016 , piece_id=MA3LBUYQ6OY7E4HMSROO5FVKJE7CEXF7SXUJIWWQX4EQCLI3OC7A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.284508037 (693h28m22.155591072s) [noise=true , address=89.149.201.147:4006 , piece_id=TQWVNQARGBZR2SRZ2LA3EVXSASEX36WXIEBGXWNWH2VQDSVC6DTA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.378387289 (693h28m22.249470334s) [noise=true , address=89.149.201.147:4015 , piece_id=SMHKGOSXQUWCO24KJLOH5RTLTZJ7MOADSCGPC6WRZUQFQHNNQ6WQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:23.055399573 (693h28m22.926482618s) [noise=true , address=81.171.14.70:4043 , piece_id=XKAAVOXN772VX5A6HSKBOU5NOYYF4U3HZ2BA5PYEOOQKCFGAIAFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:28.963038821 (693h28m28.834121857s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:32.765601718 (693h28m32.636684753s) [noise=true , address=185.167.204.205:4019 , piece_id=JKTSOUCJ2OF43PZKOKLRGUHSKRFWHQNMTIESGBK4MQNZO6VTWCLQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:36.941962859 (693h28m36.813045904s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:40.179862153 (693h28m40.050945188s) [noise=true , address=185.167.204.205:4010 , piece_id=CJBQASNMJAA5VBAFF65GU22ITWXU3RLHF6B7TFIDSUWCGSJM3OIA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:43.522593496 (693h28m43.393676531s) [noise=true , address=185.167.204.206:4041 , piece_id=QVVTZZDAAESA7IKRLWUMFMXOM4WIRUA6DP4FU2YKXSLK4WPTEC4A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:48.895462434 (693h28m48.766545479s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:58:18.486768101 (693h29m18.357851136s) [noise=true , address=185.167.204.205:4025 , piece_id=OFNEI5EOXKHZNLNDSJUQSMOBYUIGCFSAVLKRSJ5TKB3KX6ZXGN5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1cnhWXZCgZz567E5PAaPyfNKkmsSa4imaWPmfW2ir7U48gYEkp, address:185.167.204.205:4025): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487034448 (693h29m18.358117483s) [noise=true , address=185.167.204.206:4047 , piece_id=ZHLGMV427BQKUCYXOXRSMGVYTQH5TR7H62BKMOHY6QZHFQLCY2BQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487255509 (693h29m18.358338554s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12hK2MSVcFNaCVp7k4krUva7jXEJYhjnHPxPrMgq87jcDXJPkv2, address:185.167.204.206:4051): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487444569 (693h29m18.358527604s) [noise=true , address=185.167.204.205:4027 , piece_id=VIHPQE3Q7CRTAVPIYKTT4PIRU3RW4LPLUUWULRT66IX574KZ2GBA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487563535 (693h29m18.35864657s) [noise=true , address=185.167.204.205:4011 , piece_id=QBDCX7YUNQXSNX5BNO62KY4EZF7Y24KSMGDVR3XMBZF4JVM4C5FQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: read tcp 188.166.225.170:40296->185.167.204.205:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487627026 (693h29m18.358710072s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: read tcp 188.166.225.170:45010->185.167.204.205:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487692902 (693h29m18.358775937s) [seg_pos={0 0}, upload=607286]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil> Jan 3 12:58:18.668736929 (693h29m18.539819974s) [upload=607286]: Done uploading segments... Jan 3 12:58:18.668774270 (693h29m18.539857315s) [upload=607286]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22) Duration: 19.416021198s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXX.X...X...XX.X.X.XXX.XXXXX..XX...X.X.X..XX.XX] duration: 1m0.004445704s trace: https://jaeger.dev.storj.io/trace/7d8138b5fa5475fd FAIL (2026-01-03 12:58:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, Piece ID: SWZXXHPN62PRR6TWOWMKYNOLGMAOVJVOAVVAXJ7E3CDGR4BTUYTQ) context deadline exceeded; piecestore: (Node ID: 1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, Piece ID: JI2AVFGVXURAMEMD6KOYXWSKAXULGTO44BQXTBZZDOGWWSEMGZ7Q) context deadline exceeded; piecestore: (Node ID: 1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, Piece ID: 2AYVP3IAABEIZ4Q2KBSAI6GRL5PK2KKPQMLS6THC3NUIASPH2VRQ) context deadline exceeded; piecestore: (Node ID: 12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, Piece ID: B4OVT3HYCZP6OITPKUOQMMFFR732VE26FAVARCPDHNWT6BODUE6A) context deadline exceeded ======LOGS====== rue , address=64.49.13.27:4965 , piece_id=QO6KZMMSWRFV4CI3MLG2K4LGU7ED5HNFB4O3GSKVGPO26ERBK27A, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.147669919 (693h28m35.018752964s) [noise=true , address=145.79.227.3:4183 , piece_id=WJPA24D7URDUI7ISBEFYQT7H7YFNC5AAJDM6MEVO5QLVJEOYBALA, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.148193747 (693h28m35.019276782s) [noise=true , address=145.79.227.49:4369 , piece_id=O47A7ZMFS3PCZZIUZJWOGZZ7PUFUHOZ4EVKJ7YXS5D4X6IDZUUXQ, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.148203465 (693h28m35.01928651s) [noise=true , address=145.79.227.40:4071 , piece_id=O4RT47D7M47CNKFQUGM4UHK6FMD64MUMFTPP2U74OFRRHT6OCY6A, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.148721041 (693h28m35.019804076s) [noise=true , address=145.79.227.33:4875 , piece_id=3YY7PIX6XUELVKVDBYMIONI4LY6GYFTOOWHHBUOLKFJ5K3OPHQZA, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.149151370 (693h28m35.020234405s) [noise=true , address=77.90.31.15:4725 , piece_id=2AYVP3IAABEIZ4Q2KBSAI6GRL5PK2KKPQMLS6THC3NUIASPH2VRQ, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.150823273 (693h28m35.021906308s) [noise=true , address=145.79.227.34:4919 , piece_id=DKUDH5NDQM27NCHXXB3VUX5EYQSPTONGPFLVFM75LH5KK3BFLUDA, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.151160576 (693h28m35.022243621s) [noise=true , address=145.79.227.44:4213 , piece_id=VIZW2HDZSKBITSBKACY6QBQRALI3Q6D5LWIABYCJEPWNFFRPO4KA, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.151363322 (693h28m35.022446357s) [noise=true , address=64.49.13.169:4713 , piece_id=CQPJSDCZ6CKT557X6ELFN7LEJFJNIRK5CRP6OPJELSZEEWBB6FNA, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.151797929 (693h28m35.022880974s) [noise=true , address=145.79.227.30:4791 , piece_id=ARNVNHVF6UP7C5BE2SXNPXZIX3AY44A6HWQJ4L2TCXQF3GZZLG7Q, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.152302180 (693h28m35.023385225s) [noise=true , address=145.79.227.32:4846 , piece_id=UNLAXSLHEXAEYQRJ4VSJUJZBJIEVS6AWBNUULUHXGGTCMUHRYPLA, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.152757677 (693h28m35.023840712s) [noise=true , address=145.79.227.6:4263 , piece_id=2EZPVUJL66OXRMBDQTQMCLLDNKPJGCJMSTHL6R6MTAWZV7TU4K3Q, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.153147479 (693h28m35.024230524s) [noise=true , address=145.79.227.43:4177 , piece_id=QNC2RQ7XNNBOYRHM4QCUZ4M4IAAHSMIBY7M3UF2TT7GD4BDBRTZQ, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.153322553 (693h28m35.024405598s) [noise=true , address=77.90.31.10:4548 , piece_id=SWZXXHPN62PRR6TWOWMKYNOLGMAOVJVOAVVAXJ7E3CDGR4BTUYTQ, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:35.154457163 (693h28m35.025540209s) [noise=true , address=145.79.227.35:4500 , piece_id=VIB7UQ73FUN6CZGPQFD72QFXORYLBGYVGVNDKV2V3GNNRERIE5IQ, seg_pos={0 0}, upload=607291]: Uploading piece... Jan 3 12:57:36.554960913 (693h28m36.426043949s) [upload=607291]: Next returned nil segment Jan 3 12:57:36.554981663 (693h28m36.426064698s) [upload=607291]: Waiting for error group managing segments... Jan 3 12:57:36.913718437 (693h28m36.784801472s) [noise=true , address=64.49.13.154:4927 , piece_id=M6GNODP2ZEN4LDYPOYNRSNU4UYVAUTJAP3J6IA2ZIFNKWABVN3YA, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:37.005264156 (693h28m36.876347211s) [noise=true , address=64.49.13.27:4965 , piece_id=QO6KZMMSWRFV4CI3MLG2K4LGU7ED5HNFB4O3GSKVGPO26ERBK27A, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:37.417931433 (693h28m37.289014468s) [noise=true , address=45.91.116.13:4182 , piece_id=JI2AVFGVXURAMEMD6KOYXWSKAXULGTO44BQXTBZZDOGWWSEMGZ7Q, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:38.355146314 (693h28m38.226229349s) [noise=true , address=77.90.31.15:4725 , piece_id=2AYVP3IAABEIZ4Q2KBSAI6GRL5PK2KKPQMLS6THC3NUIASPH2VRQ, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:39.405160824 (693h28m39.276243869s) [noise=true , address=145.79.227.32:4846 , piece_id=UNLAXSLHEXAEYQRJ4VSJUJZBJIEVS6AWBNUULUHXGGTCMUHRYPLA, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:39.443258368 (693h28m39.314341403s) [noise=true , address=145.79.227.33:4875 , piece_id=3YY7PIX6XUELVKVDBYMIONI4LY6GYFTOOWHHBUOLKFJ5K3OPHQZA, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:39.492577300 (693h28m39.363660335s) [noise=true , address=77.90.31.10:4548 , piece_id=SWZXXHPN62PRR6TWOWMKYNOLGMAOVJVOAVVAXJ7E3CDGR4BTUYTQ, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:39.605169123 (693h28m39.476252168s) [noise=true , address=145.79.227.34:4919 , piece_id=DKUDH5NDQM27NCHXXB3VUX5EYQSPTONGPFLVFM75LH5KK3BFLUDA, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:39.657339664 (693h28m39.528422709s) [noise=true , address=145.79.227.30:4791 , piece_id=ARNVNHVF6UP7C5BE2SXNPXZIX3AY44A6HWQJ4L2TCXQF3GZZLG7Q, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:40.012092623 (693h28m39.883175658s) [noise=true , address=145.79.227.43:4177 , piece_id=QNC2RQ7XNNBOYRHM4QCUZ4M4IAAHSMIBY7M3UF2TT7GD4BDBRTZQ, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:40.042651031 (693h28m39.913734066s) [noise=true , address=145.79.227.35:4500 , piece_id=VIB7UQ73FUN6CZGPQFD72QFXORYLBGYVGVNDKV2V3GNNRERIE5IQ, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:40.416573100 (693h28m40.287656145s) [noise=true , address=145.79.227.40:4071 , piece_id=O4RT47D7M47CNKFQUGM4UHK6FMD64MUMFTPP2U74OFRRHT6OCY6A, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:41.369401362 (693h28m41.240484397s) [noise=true , address=145.79.227.6:4263 , piece_id=2EZPVUJL66OXRMBDQTQMCLLDNKPJGCJMSTHL6R6MTAWZV7TU4K3Q, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:41.622565879 (693h28m41.493648914s) [noise=true , address=77.90.31.18:4830 , piece_id=B4OVT3HYCZP6OITPKUOQMMFFR732VE26FAVARCPDHNWT6BODUE6A, seg_pos={0 0}, upload=607291]: Done uploading piece. err: <nil> Jan 3 12:57:41.622583362 (693h28m41.493666397s) [seg_pos={0 0}, upload=607291]: Segment reached optimal threshold of 14 pieces. Jan 3 12:57:41.622763886 (693h28m41.493846921s) [noise=true , address=64.49.13.169:4713 , piece_id=CQPJSDCZ6CKT557X6ELFN7LEJFJNIRK5CRP6OPJELSZEEWBB6FNA, seg_pos={0 0}, upload=607291]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:33744->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:33744->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33744->64.49.13.169:4713: use of closed network connection Jan 3 12:57:41.622769748 (693h28m41.493852783s) [noise=true , address=64.49.13.169:4713 , piece_id=CQPJSDCZ6CKT557X6ELFN7LEJFJNIRK5CRP6OPJELSZEEWBB6FNA, seg_pos={0 0}, upload=607291]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.623087933 (693h28m41.494170968s) [noise=true , address=145.79.227.44:4213 , piece_id=VIZW2HDZSKBITSBKACY6QBQRALI3Q6D5LWIABYCJEPWNFFRPO4KA, seg_pos={0 0}, upload=607291]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g): protocol: read tcp 188.166.225.170:43184->145.79.227.44:4213: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:57:41.623094917 (693h28m41.494177962s) [noise=true , address=145.79.227.44:4213 , piece_id=VIZW2HDZSKBITSBKACY6QBQRALI3Q6D5LWIABYCJEPWNFFRPO4KA, seg_pos={0 0}, upload=607291]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.623145202 (693h28m41.494228247s) [noise=true , address=145.79.227.49:4369 , piece_id=O47A7ZMFS3PCZZIUZJWOGZZ7PUFUHOZ4EVKJ7YXS5D4X6IDZUUXQ, seg_pos={0 0}, upload=607291]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.623165200 (693h28m41.494248245s) [noise=true , address=145.79.227.49:4369 , piece_id=O47A7ZMFS3PCZZIUZJWOGZZ7PUFUHOZ4EVKJ7YXS5D4X6IDZUUXQ, seg_pos={0 0}, upload=607291]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.623287995 (693h28m41.49437103s) [noise=true , address=145.79.227.3:4183 , piece_id=WJPA24D7URDUI7ISBEFYQT7H7YFNC5AAJDM6MEVO5QLVJEOYBALA, seg_pos={0 0}, upload=607291]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ): protocol: read tcp 188.166.225.170:48516->145.79.227.3:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:57:41.623295038 (693h28m41.494378083s) [noise=true , address=145.79.227.3:4183 , piece_id=WJPA24D7URDUI7ISBEFYQT7H7YFNC5AAJDM6MEVO5QLVJEOYBALA, seg_pos={0 0}, upload=607291]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.623319373 (693h28m41.494402419s) [seg_pos={0 0}, upload=607291]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 3 12:57:41.623393526 (693h28m41.494476561s) [upload=607291]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:57:41.623398836 (693h28m41.494481871s) [upload=607291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:57:41.907551419 (693h28m41.778634464s) [upload=607291]: Done uploading segments... Jan 3 12:57:41.907572769 (693h28m41.778655804s) [upload=607291]: Upload finished. err: <nil> Duration: 1m0.004457908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................X.X........X.............X..............] duration: 18.275142688s trace: https://jaeger.dev.storj.io/trace/165637e999409e3 FAIL (2026-01-03 12:58:34 UTC) orbiter warning: context deadline exceeded Duration: 18.275173316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..........X.................X..................] duration: 7.836871328s trace: https://jaeger.dev.storj.io/trace/640fd8ac8dea8303 FAIL (2026-01-03 12:58:28 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 7.836891786s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.754649096s trace: https://jaeger.dev.storj.io/trace/3c6be275344fe875 FAIL (2026-01-03 12:58:23 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.754696877s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........X.................X.................] duration: 26.778184139s trace: https://jaeger.dev.storj.io/trace/640fd8ac8dea8303 FAIL (2026-01-03 12:58:19 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 26.778216191s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 10 out of 60 is failed (threshold=3) [.......X..X.....X...X................X.....X..X...X.X......X] duration: 1m0.184775077s trace: https://jaeger.dev.storj.io/trace/53bc30794fd42896 FAIL (2026-01-03 12:58:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 22) ======LOGS====== 286]: Uploading piece... Jan 3 12:57:19.136747473 (693h28m19.007830508s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137034108 (693h28m19.008117143s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137189855 (693h28m19.008272891s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137413200 (693h28m19.008496245s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137560532 (693h28m19.008643567s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.137902893 (693h28m19.008985938s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138193637 (693h28m19.009276692s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138313765 (693h28m19.0093968s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:19.138696524 (693h28m19.009779569s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Uploading piece... Jan 3 12:57:20.027400153 (693h28m19.898483188s) [upload=607286]: Next returned nil segment Jan 3 12:57:20.027453063 (693h28m19.898536108s) [upload=607286]: Waiting for error group managing segments... Jan 3 12:57:20.410138690 (693h28m20.281221725s) [noise=true , address=185.167.204.205:4002 , piece_id=MOWRJZ3WLWZZXXB5ENPBOVZGVAD2V353VZ5CWCEO22PPZQGROVHQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:20.492391803 (693h28m20.363474848s) [noise=true , address=185.167.204.205:4022 , piece_id=CEGF62VNP7MZO7WADY2WL7WZX6UDOUA5NTWGGODTJGNKHBCCGJVA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.019412959 (693h28m20.890496004s) [noise=true , address=81.171.14.70:4030 , piece_id=GDSHOILKCEFT5BP2R6ZZDUQ5DF4CO5H3OD754ROW6GFRU3UU35TA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.054989251 (693h28m20.926072287s) [noise=true , address=81.171.14.70:4045 , piece_id=VFP3343XBNUM3CWA4ZXGUHIXYFKLPKR5G5HT3QK54QVYNLVF46AQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.057129006 (693h28m20.928212052s) [noise=true , address=81.171.14.70:4028 , piece_id=P42SQIYXEJJKQEI2GFXRU5JIJ5G3XHLT2SM45DCE67E7S5TSPSYA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.060624582 (693h28m20.931707617s) [noise=true , address=81.171.14.70:4048 , piece_id=AFKS335Y3NNXPTUMDLVVJLU5SWSW33YNXUIZ2QVE2OMQ7T2YHENA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.066618325 (693h28m20.93770136s) [noise=true , address=81.171.14.70:4047 , piece_id=AENJUR6444XL7UVU2D7L72SAULAT3FSQ5H76JEJN3TBGNRNZLPFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.076354184 (693h28m20.947437219s) [noise=true , address=81.171.14.70:4046 , piece_id=CXZXC744XXEYRZ252W2JM5N3VXFWA6UHUQYM6Z5WME7Q2B66YS5Q, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.084082891 (693h28m20.955165936s) [noise=true , address=81.171.14.70:4036 , piece_id=2HJT4WVPGHYRN2AEGR2RIL522S3XQX6AWLMH3IHKHMGNOW5HAGAQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.565324133 (693h28m21.436407168s) [noise=true , address=81.171.14.70:4032 , piece_id=5X2IPF6QMSEKVGMZFXBBW3DVXAK5SD3ZELAGWPRFN5C6SEVNUNWA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.584630255 (693h28m21.455713291s) [noise=true , address=81.171.14.70:4038 , piece_id=JI4QF46ZDL3TM4EFESCJSFFKIYRTF7CJZCC2HMUQZGO4CP2M4V5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.587294258 (693h28m21.458377303s) [noise=true , address=81.171.14.70:4034 , piece_id=GAFESRFTUPAOIHJ5LKEFK7AIZMLKL7JO5NCC26ZR27PQTTYH5UUA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:21.874175286 (693h28m21.745258321s) [noise=true , address=89.149.201.147:4016 , piece_id=MA3LBUYQ6OY7E4HMSROO5FVKJE7CEXF7SXUJIWWQX4EQCLI3OC7A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.284508037 (693h28m22.155591072s) [noise=true , address=89.149.201.147:4006 , piece_id=TQWVNQARGBZR2SRZ2LA3EVXSASEX36WXIEBGXWNWH2VQDSVC6DTA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:22.378387289 (693h28m22.249470334s) [noise=true , address=89.149.201.147:4015 , piece_id=SMHKGOSXQUWCO24KJLOH5RTLTZJ7MOADSCGPC6WRZUQFQHNNQ6WQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:23.055399573 (693h28m22.926482618s) [noise=true , address=81.171.14.70:4043 , piece_id=XKAAVOXN772VX5A6HSKBOU5NOYYF4U3HZ2BA5PYEOOQKCFGAIAFA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:28.963038821 (693h28m28.834121857s) [noise=true , address=185.167.204.206:4046 , piece_id=3IYDX2RTM32XUJVB5AFMABZY7FUPNOCJZ4YAAM7RDBEIS3WBY7GQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:32.765601718 (693h28m32.636684753s) [noise=true , address=185.167.204.205:4019 , piece_id=JKTSOUCJ2OF43PZKOKLRGUHSKRFWHQNMTIESGBK4MQNZO6VTWCLQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:36.941962859 (693h28m36.813045904s) [noise=true , address=185.167.204.206:4066 , piece_id=ROSZF5K5N4JIF6QGWWUTXNROLU7CM7565SFVBPPFQBTOCHHAXSAA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:40.179862153 (693h28m40.050945188s) [noise=true , address=185.167.204.205:4010 , piece_id=CJBQASNMJAA5VBAFF65GU22ITWXU3RLHF6B7TFIDSUWCGSJM3OIA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:43.522593496 (693h28m43.393676531s) [noise=true , address=185.167.204.206:4041 , piece_id=QVVTZZDAAESA7IKRLWUMFMXOM4WIRUA6DP4FU2YKXSLK4WPTEC4A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:57:48.895462434 (693h28m48.766545479s) [noise=true , address=185.167.204.205:4026 , piece_id=Y4Y6U5TUT54UZOSCBLA6SWG2O27U2F7U4RLEJQR4P3TD7RIHVT5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: <nil> Jan 3 12:58:18.486768101 (693h29m18.357851136s) [noise=true , address=185.167.204.205:4025 , piece_id=OFNEI5EOXKHZNLNDSJUQSMOBYUIGCFSAVLKRSJ5TKB3KX6ZXGN5A, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1cnhWXZCgZz567E5PAaPyfNKkmsSa4imaWPmfW2ir7U48gYEkp, address:185.167.204.205:4025): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487034448 (693h29m18.358117483s) [noise=true , address=185.167.204.206:4047 , piece_id=ZHLGMV427BQKUCYXOXRSMGVYTQH5TR7H62BKMOHY6QZHFQLCY2BQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487255509 (693h29m18.358338554s) [noise=true , address=185.167.204.206:4051 , piece_id=JAQWQZY4275I3VF2TIJAW55V3BFSJTWJRDJOKAOXUB4JVGJQURPQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12hK2MSVcFNaCVp7k4krUva7jXEJYhjnHPxPrMgq87jcDXJPkv2, address:185.167.204.206:4051): protocol: expected piece hash; context deadline exceeded; EOF Jan 3 12:58:18.487444569 (693h29m18.358527604s) [noise=true , address=185.167.204.205:4027 , piece_id=VIHPQE3Q7CRTAVPIYKTT4PIRU3RW4LPLUUWULRT66IX574KZ2GBA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487563535 (693h29m18.35864657s) [noise=true , address=185.167.204.205:4011 , piece_id=QBDCX7YUNQXSNX5BNO62KY4EZF7Y24KSMGDVR3XMBZF4JVM4C5FQ, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: read tcp 188.166.225.170:40296->185.167.204.205:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487627026 (693h29m18.358710072s) [noise=true , address=185.167.204.205:4009 , piece_id=JMDQCBPV6EA35OQRUQQPRN45NVTOW5E2UH6EA3L5F6ELQVHYS2PA, seg_pos={0 0}, upload=607286]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: read tcp 188.166.225.170:45010->185.167.204.205:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:58:18.487692902 (693h29m18.358775937s) [seg_pos={0 0}, upload=607286]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil> Jan 3 12:58:18.668736929 (693h29m18.539819974s) [upload=607286]: Done uploading segments... Jan 3 12:58:18.668774270 (693h29m18.539857315s) [upload=607286]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22) Duration: 1m0.184800425s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXX.X.XXXXXXX.XXXX.XXX.XX.XX....X.XXXXXX] duration: 5.00169899s trace: https://jaeger.dev.storj.io/trace/2f55ad33e8e8e22 FAIL (2026-01-03 12:58:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00172071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [..X...X.XX....XX.XXX.XXX.XXXX.XXXXX.XX.XXXXXX.XX.XXX.X.XX.X.] duration: 24.287584667s trace: https://jaeger.dev.storj.io/trace/11d904e2db0182d9 FAIL (2026-01-03 12:58:12 UTC) orbiter warning: context deadline exceeded Duration: 24.28761291s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 12.226180204s trace: https://jaeger.dev.storj.io/trace/5a0aed6ab18c8274 FAIL (2026-01-03 12:58:07 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 1F3yVaqH52tef8gWWe4JAWh47yRjDDEqYyou7abzPnDkDEeson, Piece ID: FEJXLLJSEBSH4XFEJX54FSB3U3237HMWQWZJAIGHDRFQHPNB5KLA) context deadline exceeded; piecestore: (Node ID: 1yyv7LXhZGJFvkTV4vafuZ6KrQkBWB7zH7q2emQ3dNat8iuWsK, Piece ID: OWDX5T5OCZF6L7DTGORIV24KCTBVMI5NUWJC5JZVGYOR36KS3DPQ) context deadline exceeded; piecestore: (Node ID: 12q1oXSNGG6uCuHBtvGTxrbWLnchY11hmhfbiRzcm1zCwGvz7Zh, Piece ID: T4E3TYY6CEPW54CARC2EIJS6YF4CDABBI7LIJQX5Q6AVFO7JEVNQ) context deadline exceeded; piecestore: (Node ID: 1UtnhDdcRHT4L5h1MxGCRWfY24bCETpRgMz62K8s4qAMkwS3fG, Piece ID: 64IAPWLKLFHF7MI3IAHFZHBNOTH64G4OPBC2NGPL5DZCWJHBXOFQ) context deadline exceeded; piecestore: (Node ID: 1x29ZBapLNUJt9E9WjyjUYav5LK4d3h5MsmZAVH2L2daqiANw1, Piece ID: QPZFZ3N4E2VSRMUB7XU6LZUH4WJFVEFYQMLTZ6O7ACG5RMBHNDJA) context deadline exceeded; piecestore: (Node ID: 1EQTE1zDuMWpoXmTB2jkH9cv2Pe2yAGWCc6A4iNznkSKzkN8pM, Piece ID: SAYUQDKORTF7IF7XHIUXCI454RUHCCBQPCIBKMX6T2E6LR2DRRUA) context deadline exceeded; piecestore: (Node ID: 1o2qgqfNpCnXxd66kmxTTqeQzWzuegmzrhqkP9bZ9CXmjLzva5, Piece ID: Q5AR3ZKTR6TX2ZQAH34DDVLT7IP5Y5N4HKSHNJPJQMS6QNTRGXIA) context deadline exceeded; piecestore: (Node ID: 12a3qUuetnZPo3mAGDWaPQtWQe8v5AveuKqscgS3aH1N1VMyPjV, Piece ID: 3XFFIFZM3UOPDOSWPVHXJQBSUZZ273XG7IJ25H6Z77U5R5KBK5YQ) context deadline exceeded; piecestore: (Node ID: 12bGxp1vLoht7Be8UHdfPXW7z4g51HbLw9fDKRa9jgv1ebztzJC, Piece ID: 3H3LTDGAPSS6AIEQ4A3QPXUZTSQIUJO7WDZC2AUAUZNVFVMB5HXQ) context deadline exceeded; piecestore: (Node ID: 12LDFuZbL93YbimH8WpmMgmajNhpBnLZ1bMB7K7KW9WyNKYV6Zw, Piece ID: YHTBHCGJX5UREKEVI64QA6WJGLIUIJU7S52M5VOLBESFGFYEW3AA) context deadline exceeded; piecestore: (Node ID: 12MiSWqqubBGaQm54btriUNz2FqoLjjEgNakbxRQ2MLXWtDiqu6, Piece ID: IQXPA7FCB7WJFUI5XDCH3CYWJK3RB6YWBQFHNMPZNC7U6ZK3JTRQ) context deadline exceeded Duration: 12.226197176s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X..XX.X..X.XX.X.....XXX.XXXX..X..X...X..XX..XX..X...XX..XX.X] duration: 30.004777976s trace: https://jaeger.dev.storj.io/trace/6384b51b065162e6 FAIL (2026-01-03 12:58:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: 4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA) context deadline exceeded; piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: 3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA) context deadline exceeded ======LOGS====== Jan 3 12:57:32.139975894 (693h28m32.011058929s) [upload=607290]: Starting upload Jan 3 12:57:32.140035398 (693h28m32.011118433s) [upload=607290]: Done starting upload Jan 3 12:57:32.140163461 (693h28m32.011246496s) [upload=607290]: Uploading segments... Jan 3 12:57:32.140286756 (693h28m32.011369801s) [upload=607290]: Got next segment. Inline: false Jan 3 12:57:32.140334296 (693h28m32.011417331s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 3 12:57:32.140339225 (693h28m32.01142226s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 3 12:57:32.386239220 (693h28m32.257322265s) [seg_pos={0 0}, upload=607290]: Begin upload segment... Jan 3 12:57:32.386291079 (693h28m32.257374114s) [seg_pos={0 0}, upload=607290]: Done begin upload segment. Jan 3 12:57:32.386337226 (693h28m32.257420261s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386347246 (693h28m32.257430281s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386399996 (693h28m32.257483041s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.386793715 (693h28m32.257876751s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387239204 (693h28m32.258322239s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.387746029 (693h28m32.258829065s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.388247554 (693h28m32.259330619s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.390590996 (693h28m32.261674041s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.394182876 (693h28m32.265265912s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.397175685 (693h28m32.26825872s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.398532489 (693h28m32.269615534s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.400607720 (693h28m32.271690755s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.402897280 (693h28m32.273980315s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.403308622 (693h28m32.274391657s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.405982995 (693h28m32.27706603s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:32.407419360 (693h28m32.278502395s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Uploading piece... Jan 3 12:57:34.297376941 (693h28m34.168459976s) [upload=607290]: Next returned nil segment Jan 3 12:57:34.297397681 (693h28m34.168480716s) [upload=607290]: Waiting for error group managing segments... Jan 3 12:57:34.931863919 (693h28m34.802946954s) [noise=true , address=45.91.116.7:4011 , piece_id=2VLATTMFAUNQTO5DTOUDOUUWWJDPYBOYLTMLMRJRKPILZW6MEOIA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.117922626 (693h28m34.989005671s) [noise=true , address=77.90.31.16:4735 , piece_id=LYKLVN5N7XXXKLU3JXCXATV4CXAT2DKYIPHIXQF32FWWRO7OKI2A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.556098205 (693h28m35.42718125s) [noise=true , address=145.79.227.48:4302 , piece_id=4P3NRHGJAI4AQOJJYIBU6LACHU4L7NHXTQPRNSHZEUZ56UNDKSGA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:35.595938106 (693h28m35.467021141s) [noise=true , address=145.79.227.3:4151 , piece_id=3MZVTJJ25OOXB7JPJEEHZHNPEN6T4AWZKZ6CPR4AEBMIWFSNPMWA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.211248298 (693h28m36.082331333s) [noise=true , address=145.79.227.50:4375 , piece_id=PA44VVUBGFN6VEZ6S36WOAB3KCWDLP3PHFWKPDUX2OXCQ4ZMNKRA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.430239493 (693h28m36.301322528s) [noise=true , address=145.79.227.51:4409 , piece_id=JSK7M55OK3K4J53XOJKSGL7GYXCKBPHDM6A5ONQUJISM4KFHMHBQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:36.787016996 (693h28m36.658100031s) [noise=true , address=145.79.227.14:4837 , piece_id=AFIUVHTMNZD6HLPRFL2VG2TOFGLMYE562GX3RS254EO2MAFBGN6A, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.437957986 (693h28m37.309041031s) [noise=true , address=145.79.227.46:4603 , piece_id=UBHRGQTQKBKP7AMRA6APYGIIDCJV2YKXHWMHFBEIB5GQO6MQODFA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:37.960941243 (693h28m37.832024278s) [noise=true , address=45.91.116.13:4153 , piece_id=V2OIBCRJ5TQ4K7HFD4LBFMUCV7SIPXH5EOSX4O5NK24NQQBLZJZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:38.634528192 (693h28m38.505611227s) [noise=true , address=145.79.227.23:4545 , piece_id=BFM3C6U64YNCPPILJNVDKFNOBECPMQ33MHIPYQNTTAA7AWPTTWVA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.241920683 (693h28m40.113003718s) [noise=true , address=145.79.227.15:4876 , piece_id=WQQREOXQGJWGEYFTWVEPWOATJCCK6MPEYRFZU6P3J7N3JJTMA7WQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:40.826824296 (693h28m40.697907331s) [noise=true , address=145.79.227.12:4765 , piece_id=ULPWMZI57DAN3K3DVLFYLA64OFNPQA5FJWLJUJAADYYDNFQ7GTWQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612734057 (693h28m41.483817092s) [noise=true , address=77.90.31.11:4554 , piece_id=I2SFRKIGFJKNJXSNEP7WMQNKC2LLMW2L2AE6C5ADBNAR4P7WKAXA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: <nil> Jan 3 12:57:41.612755909 (693h28m41.483838954s) [seg_pos={0 0}, upload=607290]: Segment reached optimal threshold of 13 pieces. Jan 3 12:57:41.613062503 (693h28m41.484145548s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: read tcp 188.166.225.170:48988->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:57:41.613069997 (693h28m41.484153032s) [noise=true , address=145.79.227.21:4473 , piece_id=F6PR4PW2FULJJFEEYTRT55H36HOECISKYMS2DB6GUA3BQCFTA44Q, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613316457 (693h28m41.484399492s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613324452 (693h28m41.484407487s) [noise=true , address=145.79.227.11:4731 , piece_id=DRPGDWE6F565OYI76QNN6WCI33MPCH2XXCTWQUYUL47SQVKJPNZQ, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613457245 (693h28m41.48454029s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Jan 3 12:57:41.613494676 (693h28m41.484577731s) [noise=true , address=145.79.227.6:4258 , piece_id=IPVFX73TWL6T5QLQVL35JM62QVDZIYU2JRDQRPX3PWFT5OWRTHAA, seg_pos={0 0}, upload=607290]: Long tail cancellation due to optimal threshold Jan 3 12:57:41.613539030 (693h28m41.484622075s) [seg_pos={0 0}, upload=607290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 3 12:57:41.613773797 (693h28m41.484856842s) [upload=607290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:57:41.613782373 (693h28m41.484865428s) [upload=607290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:57:41.908933460 (693h28m41.780016505s) [upload=607290]: Done uploading segments... Jan 3 12:57:41.908950523 (693h28m41.780033558s) [upload=607290]: Upload finished. err: <nil> Duration: 30.004801611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........X.................X................] duration: 11.427109151s trace: https://jaeger.dev.storj.io/trace/640fd8ac8dea8303 FAIL (2026-01-03 12:57:48 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 11.427136574s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X.....................X.....] duration: 12.570523582s trace: https://jaeger.dev.storj.io/trace/2a592caa2cd77abc FAIL (2026-01-03 12:57:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.570554048s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [...X...X.XX....XX.XXX.XXX.XXXX.XXXXX.XX.XXXXXX.XX.XXX.X.XX.X] duration: 30.001651524s trace: https://jaeger.dev.storj.io/trace/11d904e2db0182d9 FAIL (2026-01-03 12:57:45 UTC) orbiter warning: context deadline exceeded Duration: 30.001675428s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXX.X.XXXXXXX.XXXX.XXX.XX.XX....X.XXXXX] duration: 5.001258071s trace: https://jaeger.dev.storj.io/trace/560e7bcb4f9e89c2 FAIL (2026-01-03 12:57:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001283228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.X........X.............X.............] duration: 27.579027106s trace: https://jaeger.dev.storj.io/trace/165637e999409e3 FAIL (2026-01-03 12:57:38 UTC) orbiter warning: context deadline exceeded Duration: 27.579051233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXXXXXXX.XXXXX.X...X...XX.X.X.XXX.XXXXX..XX...X.X.X..XX.X] duration: 1m0.003283158s trace: https://jaeger.dev.storj.io/trace/76a6add9c91af2af FAIL (2026-01-03 12:57:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X97ZxbjkVLdJgBNb8F6ejLtm6gVGn2GxxEXFcmFh3psyFxqaw, Piece ID: L5V6AW5OH5TMN5HNK7K2DZF4UQ7G5QEWKMVY5DFQOGL5WFCFNDLQ) context deadline exceeded; piecestore: (Node ID: 1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx, Piece ID: IGW2IY5Z6XQPZ5PSL6LUOCDFIEIJ6Q2OOO3NMSFYPGOCUH3SBIJQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: 7PQ5NFWTEJA7LFCN33ANBLZ2TJ3WME4EXUF6ET36OBFDCCQ4AB3A) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: YARYZV3LIGWV35GGXS7VLLFZBE4UGBLGGMJP34EIDPV2DGCJ4HNQ) context deadline exceeded ======LOGS====== :34.959378776 (693h27m34.830461821s) [noise=true , address=77.90.31.11:4583 , piece_id=MN2XDWXSUQLNVLR75VS64TNL5PLNMOZDKJNJUQ6CRY6HMKY5LCXQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.960215509 (693h27m34.831298544s) [noise=true , address=145.79.227.30:4783 , piece_id=XKZXB3SCZI24LYNT3YCBUUP52HEPAMM735XY3POAYT7JONGK2QUA, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.960566016 (693h27m34.831649051s) [noise=true , address=145.79.227.29:4962 , piece_id=F4INWPLBB3OEHSPZQLWTKT5QREIKTE7FLF7C4HPUUIXXG2EFE4GA, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.960693209 (693h27m34.831776254s) [noise=true , address=77.90.31.14:4673 , piece_id=5BFMBVJQMEU47NUT3JAGSOR7V4YBYELAKVL64M4SQOBSK6CNGGRQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.960858183 (693h27m34.831941218s) [noise=true , address=77.90.31.18:4839 , piece_id=IGW2IY5Z6XQPZ5PSL6LUOCDFIEIJ6Q2OOO3NMSFYPGOCUH3SBIJQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.961144378 (693h27m34.832227413s) [noise=true , address=64.49.13.138:4491 , piece_id=L5V6AW5OH5TMN5HNK7K2DZF4UQ7G5QEWKMVY5DFQOGL5WFCFNDLQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.961386369 (693h27m34.832469404s) [noise=true , address=145.79.227.43:4169 , piece_id=QZWL3FGB3Y44DAK27RUSGLGNNYJSQD6OGTR5V6GG7E547VFRWWWQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.961460700 (693h27m34.832543745s) [noise=true , address=145.79.227.52:4451 , piece_id=GUG2GBO73BB72RYTJXECCBSZXHUNC7W5VV7WFNXC35WJH2VW6ILA, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.961995137 (693h27m34.833078172s) [noise=true , address=145.79.227.40:4073 , piece_id=D6LKHHUDXLE4V3NFVB6RVWRQZSCBLSG5ZAPU7E5KXRZDFR45YOKQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.962325407 (693h27m34.833408442s) [noise=true , address=64.49.13.155:4913 , piece_id=FTD6H7CT67JJ267L4764PEYICCJ62WPSLHI3PEQBBLYDRYYITZWQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.962366866 (693h27m34.833449911s) [noise=true , address=45.91.116.8:4050 , piece_id=7PQ5NFWTEJA7LFCN33ANBLZ2TJ3WME4EXUF6ET36OBFDCCQ4AB3A, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.963095413 (693h27m34.834178448s) [noise=true , address=145.79.227.44:4201 , piece_id=GW37CA54DRTRFVOB6VD5EK5GCBHGTDIBZPMFPVMMDCEWHU4TR4KQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.963921266 (693h27m34.835004311s) [noise=true , address=145.79.227.14:4848 , piece_id=CKAHJSNRJQAV2EEHNFPRZNSE4SYLXG4SAMLVUYOFJ7FJ52OPRTJQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.964467094 (693h27m34.83555014s) [noise=true , address=45.91.116.4:4966 , piece_id=575QCRY5N6WOL4PJAXVSW54DRI3DSKGQCHPXGI4A3TQBOISPDORQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.965015449 (693h27m34.836098484s) [noise=true , address=145.79.227.35:4494 , piece_id=SXK6EO74KAJ2YHKWGKMVBDCD5DCWUWQOFKRGDBFSOIAJEQOUUBZQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:34.965385474 (693h27m34.836468509s) [noise=true , address=77.90.31.10:4536 , piece_id=YARYZV3LIGWV35GGXS7VLLFZBE4UGBLGGMJP34EIDPV2DGCJ4HNQ, seg_pos={0 0}, upload=607277]: Uploading piece... Jan 3 12:56:36.413282649 (693h27m36.284365684s) [upload=607277]: Next returned nil segment Jan 3 12:56:36.413296966 (693h27m36.284380022s) [upload=607277]: Waiting for error group managing segments... Jan 3 12:56:36.826366057 (693h27m36.697449092s) [noise=true , address=145.79.227.14:4848 , piece_id=CKAHJSNRJQAV2EEHNFPRZNSE4SYLXG4SAMLVUYOFJ7FJ52OPRTJQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:36.834755172 (693h27m36.705838217s) [noise=true , address=145.79.227.52:4451 , piece_id=GUG2GBO73BB72RYTJXECCBSZXHUNC7W5VV7WFNXC35WJH2VW6ILA, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.708668197 (693h27m37.579751242s) [noise=true , address=145.79.227.49:4354 , piece_id=Q3DTGVRWS2OKLII7W3XEG4HHDGTWOL4SMOZMKHSSQ5QK6EIKRLFA, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.726474152 (693h27m37.597557197s) [noise=true , address=145.79.227.44:4201 , piece_id=GW37CA54DRTRFVOB6VD5EK5GCBHGTDIBZPMFPVMMDCEWHU4TR4KQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.747247428 (693h27m37.618330463s) [noise=true , address=145.79.227.34:4926 , piece_id=6FDPFGOVJBWL7VRN64YIHLP7G36HBXG47OXWKJWW6IFDWFIIFKEQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.806547185 (693h27m37.67763022s) [noise=true , address=145.79.227.29:4962 , piece_id=F4INWPLBB3OEHSPZQLWTKT5QREIKTE7FLF7C4HPUUIXXG2EFE4GA, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.823219562 (693h27m37.694302607s) [noise=true , address=145.79.227.30:4783 , piece_id=XKZXB3SCZI24LYNT3YCBUUP52HEPAMM735XY3POAYT7JONGK2QUA, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.824411131 (693h27m37.695494176s) [noise=true , address=145.79.227.40:4073 , piece_id=D6LKHHUDXLE4V3NFVB6RVWRQZSCBLSG5ZAPU7E5KXRZDFR45YOKQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.846543625 (693h27m37.71762666s) [noise=true , address=145.79.227.43:4169 , piece_id=QZWL3FGB3Y44DAK27RUSGLGNNYJSQD6OGTR5V6GG7E547VFRWWWQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.928501405 (693h27m37.79958445s) [noise=true , address=145.79.227.35:4494 , piece_id=SXK6EO74KAJ2YHKWGKMVBDCD5DCWUWQOFKRGDBFSOIAJEQOUUBZQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:37.954260505 (693h27m37.82534354s) [noise=true , address=77.90.31.18:4839 , piece_id=IGW2IY5Z6XQPZ5PSL6LUOCDFIEIJ6Q2OOO3NMSFYPGOCUH3SBIJQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:38.079426132 (693h27m37.950509167s) [noise=true , address=77.90.31.10:4536 , piece_id=YARYZV3LIGWV35GGXS7VLLFZBE4UGBLGGMJP34EIDPV2DGCJ4HNQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:38.500623170 (693h27m38.371706215s) [noise=true , address=64.49.13.138:4491 , piece_id=L5V6AW5OH5TMN5HNK7K2DZF4UQ7G5QEWKMVY5DFQOGL5WFCFNDLQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:39.523507420 (693h27m39.394590465s) [noise=true , address=45.91.116.8:4050 , piece_id=7PQ5NFWTEJA7LFCN33ANBLZ2TJ3WME4EXUF6ET36OBFDCCQ4AB3A, seg_pos={0 0}, upload=607277]: Done uploading piece. err: <nil> Jan 3 12:56:39.523531326 (693h27m39.394614371s) [seg_pos={0 0}, upload=607277]: Segment reached optimal threshold of 14 pieces. Jan 3 12:56:39.523799076 (693h27m39.394882131s) [noise=true , address=77.90.31.11:4583 , piece_id=MN2XDWXSUQLNVLR75VS64TNL5PLNMOZDKJNJUQ6CRY6HMKY5LCXQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Jan 3 12:56:39.523808704 (693h27m39.394891749s) [noise=true , address=77.90.31.11:4583 , piece_id=MN2XDWXSUQLNVLR75VS64TNL5PLNMOZDKJNJUQ6CRY6HMKY5LCXQ, seg_pos={0 0}, upload=607277]: Long tail cancellation due to optimal threshold Jan 3 12:56:39.524100199 (693h27m39.395183244s) [noise=true , address=64.49.13.155:4913 , piece_id=FTD6H7CT67JJ267L4764PEYICCJ62WPSLHI3PEQBBLYDRYYITZWQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Jan 3 12:56:39.524111551 (693h27m39.395194596s) [noise=true , address=64.49.13.155:4913 , piece_id=FTD6H7CT67JJ267L4764PEYICCJ62WPSLHI3PEQBBLYDRYYITZWQ, seg_pos={0 0}, upload=607277]: Long tail cancellation due to optimal threshold Jan 3 12:56:39.524242369 (693h27m39.395325414s) [noise=true , address=45.91.116.4:4966 , piece_id=575QCRY5N6WOL4PJAXVSW54DRI3DSKGQCHPXGI4A3TQBOISPDORQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: read tcp 188.166.225.170:42510->45.91.116.4:4966: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:56:39.524252219 (693h27m39.395335264s) [noise=true , address=45.91.116.4:4966 , piece_id=575QCRY5N6WOL4PJAXVSW54DRI3DSKGQCHPXGI4A3TQBOISPDORQ, seg_pos={0 0}, upload=607277]: Long tail cancellation due to optimal threshold Jan 3 12:56:39.524431159 (693h27m39.395514204s) [noise=true , address=77.90.31.14:4673 , piece_id=5BFMBVJQMEU47NUT3JAGSOR7V4YBYELAKVL64M4SQOBSK6CNGGRQ, seg_pos={0 0}, upload=607277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh): protocol: read tcp 188.166.225.170:35712->77.90.31.14:4673: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 3 12:56:39.524472688 (693h27m39.395555733s) [noise=true , address=77.90.31.14:4673 , piece_id=5BFMBVJQMEU47NUT3JAGSOR7V4YBYELAKVL64M4SQOBSK6CNGGRQ, seg_pos={0 0}, upload=607277]: Long tail cancellation due to optimal threshold Jan 3 12:56:39.524498688 (693h27m39.395581723s) [seg_pos={0 0}, upload=607277]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 3 12:56:39.524664363 (693h27m39.395747408s) [upload=607277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 3 12:56:39.524672218 (693h27m39.395755253s) [upload=607277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 3 12:56:39.818074494 (693h27m39.68915753s) [upload=607277]: Done uploading segments... Jan 3 12:56:39.818096587 (693h27m39.689179622s) [upload=607277]: Upload finished. err: <nil> Duration: 1m0.003306512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 7.258865616s trace: https://jaeger.dev.storj.io/trace/3c6be275344fe875 FAIL (2026-01-03 12:57:34 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 7.258891116s core uplink ap1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.18985213s trace: https://jaeger.dev.storj.io/trace/5a0aed6ab18c8274 FAIL (2026-01-03 12:57:15 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 1F3yVaqH52tef8gWWe4JAWh47yRjDDEqYyou7abzPnDkDEeson, Piece ID: FEJXLLJSEBSH4XFEJX54FSB3U3237HMWQWZJAIGHDRFQHPNB5KLA) context deadline exceeded; piecestore: (Node ID: 1yyv7LXhZGJFvkTV4vafuZ6KrQkBWB7zH7q2emQ3dNat8iuWsK, Piece ID: OWDX5T5OCZF6L7DTGORIV24KCTBVMI5NUWJC5JZVGYOR36KS3DPQ) context deadline exceeded; piecestore: (Node ID: 12q1oXSNGG6uCuHBtvGTxrbWLnchY11hmhfbiRzcm1zCwGvz7Zh, Piece ID: T4E3TYY6CEPW54CARC2EIJS6YF4CDABBI7LIJQX5Q6AVFO7JEVNQ) context deadline exceeded; piecestore: (Node ID: 1UtnhDdcRHT4L5h1MxGCRWfY24bCETpRgMz62K8s4qAMkwS3fG, Piece ID: 64IAPWLKLFHF7MI3IAHFZHBNOTH64G4OPBC2NGPL5DZCWJHBXOFQ) context deadline exceeded; piecestore: (Node ID: 1x29ZBapLNUJt9E9WjyjUYav5LK4d3h5MsmZAVH2L2daqiANw1, Piece ID: QPZFZ3N4E2VSRMUB7XU6LZUH4WJFVEFYQMLTZ6O7ACG5RMBHNDJA) context deadline exceeded; piecestore: (Node ID: 1EQTE1zDuMWpoXmTB2jkH9cv2Pe2yAGWCc6A4iNznkSKzkN8pM, Piece ID: SAYUQDKORTF7IF7XHIUXCI454RUHCCBQPCIBKMX6T2E6LR2DRRUA) context deadline exceeded; piecestore: (Node ID: 1o2qgqfNpCnXxd66kmxTTqeQzWzuegmzrhqkP9bZ9CXmjLzva5, Piece ID: Q5AR3ZKTR6TX2ZQAH34DDVLT7IP5Y5N4HKSHNJPJQMS6QNTRGXIA) context deadline exceeded; piecestore: (Node ID: 12a3qUuetnZPo3mAGDWaPQtWQe8v5AveuKqscgS3aH1N1VMyPjV, Piece ID: 3XFFIFZM3UOPDOSWPVHXJQBSUZZ273XG7IJ25H6Z77U5R5KBK5YQ) context deadline exceeded; piecestore: (Node ID: 12bGxp1vLoht7Be8UHdfPXW7z4g51HbLw9fDKRa9jgv1ebztzJC, Piece ID: 3H3LTDGAPSS6AIEQ4A3QPXUZTSQIUJO7WDZC2AUAUZNVFVMB5HXQ) context deadline exceeded; piecestore: (Node ID: 12LDFuZbL93YbimH8WpmMgmajNhpBnLZ1bMB7K7KW9WyNKYV6Zw, Piece ID: YHTBHCGJX5UREKEVI64QA6WJGLIUIJU7S52M5VOLBESFGFYEW3AA) context deadline exceeded; piecestore: (Node ID: 12MiSWqqubBGaQm54btriUNz2FqoLjjEgNakbxRQ2MLXWtDiqu6, Piece ID: IQXPA7FCB7WJFUI5XDCH3CYWJK3RB6YWBQFHNMPZNC7U6ZK3JTRQ) context deadline exceeded Duration: 6.189877617s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)