Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.165478361s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:12:10 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.165499561s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 7.410282373s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:12:08 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.410311408s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.862950168s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:12:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.862980077s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X..............XX.....XXXX.X....XX...XXX.XX..X.XXX.XXXXXXX] duration: 5.00221046s trace: https://jaeger.dev.storj.io/trace/587c68989c34b16d FAIL (2025-12-05 13:11:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00223566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.210024096s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.21005302s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 7.2378543s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:11:26 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.237884648s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.135263279s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:11:04 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.135300329s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 7.370273164s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:10:56 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.370304835s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.156237978s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:10:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.156266162s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...X..............XX.....XXXX.X....XX...XXX.XX..X.XXX.XXXXXX] duration: 5.001347473s trace: https://jaeger.dev.storj.io/trace/4eb8d187266c1894 FAIL (2025-12-05 13:10:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001444137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.201021511s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:10:29 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.20107856s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 7.640878714s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:10:09 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.640915094s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.15610936s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:09:59 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.156139488s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [....X..............XX.....XXXX.X....XX...XXX.XX..X.XXX.XXXXX] duration: 5.001308051s trace: https://jaeger.dev.storj.io/trace/4ec02156f54fc9c7 FAIL (2025-12-05 13:09:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001334051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.345536296s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:09:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.345557645s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.959219536s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:09:38 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 4.959247819s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 7.19603269s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:09:37 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.196137911s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.....X..............XX.....XXXX.X....XX...XXX.XX..X.XXX.XXXX] duration: 5.001755232s trace: https://jaeger.dev.storj.io/trace/3da09944a54fc644 FAIL (2025-12-05 13:09:22 UTC) orbiter warning: context deadline exceeded Duration: 5.001778837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.494485353s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:09:02 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.49451577s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 9.253613861s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:08:41 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 9.253647756s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.021987689s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:08:41 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.022019712s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.191084331s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:08:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.191114808s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 8.158295302s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:08:19 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 8.158328475s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [......X..............XX.....XXXX.X....XX...XXX.XX..X.XXX.XXX] duration: 5.001365159s trace: https://jaeger.dev.storj.io/trace/20471880a6ea3eca FAIL (2025-12-05 13:08:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001392853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.306570687s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:07:36 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.306618328s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 7.30335856s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:07:34 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.303464101s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.774233879s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:07:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.774262522s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.......X..............XX.....XXXX.X....XX...XXX.XX..X.XXX.XX] duration: 5.00112992s trace: https://jaeger.dev.storj.io/trace/46e242155320c886 FAIL (2025-12-05 13:07:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001155768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.168505105s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:06:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.168536205s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.325370055s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:06:36 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.325398399s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [........X..............XX.....XXXX.X....XX...XXX.XX..X.XXX.X] duration: 5.001337888s trace: https://jaeger.dev.storj.io/trace/54feac76eb1d4851 FAIL (2025-12-05 13:06:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001363456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 6.960397081s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:06:21 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 6.960466916s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.173006537s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:05:51 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.173037927s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.........X..............XX.....XXXX.X....XX...XXX.XX..X.XXX.] duration: 5.000875868s trace: https://jaeger.dev.storj.io/trace/28587a40ef202807 FAIL (2025-12-05 13:05:45 UTC) orbiter warning: context deadline exceeded Duration: 5.000933448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 7.388988227s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:05:43 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.389022402s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.802608887s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:05:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.802640608s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.058925496s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:05:14 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.058958678s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 7.302429902s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:04:49 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.302488683s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.690130339s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:04:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.690155867s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..........X..............XX.....XXXX.X....XX...XXX.XX..X.XXX] duration: 5.001532889s trace: https://jaeger.dev.storj.io/trace/28587a40ef202807 FAIL (2025-12-05 13:04:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001649912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.780544115s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:04:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.780579552s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.504251191s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:04:06 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.504287339s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 7.355506411s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:03:46 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.355536257s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.16034727s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:03:46 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.160377307s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...........X..............XX.....XXXX.X....XX...XXX.XX..X.XX] duration: 5.000796156s trace: https://jaeger.dev.storj.io/trace/694d68f8caffabcf FAIL (2025-12-05 13:03:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00081981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [............X..............XX.....XXXX.X....XX...XXX.XX..X.X] duration: 5.000990094s trace: https://jaeger.dev.storj.io/trace/1cc164ef2b550553 FAIL (2025-12-05 13:03:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001014931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.360373157s trace: https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb FAIL (2025-12-05 13:02:57 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51) Duration: 5.360399938s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.818729283s trace: https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7 FAIL (2025-12-05 13:02:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece... Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments... Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil> Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments... Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.818763429s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X............X..............XX.....XXXX.X....XX...XXX.XX..X.] duration: 5.001752275s trace: https://jaeger.dev.storj.io/trace/238981efb5cadb3c FAIL (2025-12-05 13:02:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001783784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 7.16023165s trace: https://jaeger.dev.storj.io/trace/275a4a7f39135e2a FAIL (2025-12-05 13:02:45 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 7.16026266s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)