Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.X] duration: 30.0037013s trace: https://jaeger.dev.storj.io/trace/6b73ce31b124f7a5 FAIL (2025-12-30 12:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: U3MW7UK4PL5NLWDDEO2BNOR2ACQHIICJBSTDKJFEQGFPQBK6H47A) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 6SRRWZZKREJX5OPAYWOIA2WQDEKU2IIWS4I7RD47BFQRCPVABRDQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: BUSX6WY6TEPFJCCUU7JSDRDQU577343GAYRPI4WY7Y4HK3RJA6IQ) context deadline exceeded ======LOGS====== ]: Starting upload Dec 30 12:11:19.623004378 (596h42m19.494087433s) [upload=523063]: Done starting upload Dec 30 12:11:19.623243263 (596h42m19.494326318s) [upload=523063]: Uploading segments... Dec 30 12:11:19.623356659 (596h42m19.494439694s) [upload=523063]: Got next segment. Inline: false Dec 30 12:11:19.623395322 (596h42m19.494478367s) [upload=523063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:11:19.623402016 (596h42m19.494485071s) [upload=523063]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:11:19.868235617 (596h42m19.739318652s) [seg_pos={0 0}, upload=523063]: Begin upload segment... Dec 30 12:11:19.868305981 (596h42m19.739389026s) [seg_pos={0 0}, upload=523063]: Done begin upload segment. Dec 30 12:11:19.868312473 (596h42m19.739395508s) [noise=true , address=145.79.227.42:4123 , piece_id=6M4U2HP4OI63YLS5K243WSEPUUJXDH6UQ3IKANH4WZZADSDPFKPQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.868344514 (596h42m19.739427559s) [noise=true , address=145.79.227.41:4087 , piece_id=FIPQYMLA6VDUNYASNOTPYM2ABXLDQYC7UFCR2GV7YLQ6D7HDVK6Q, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.868834597 (596h42m19.739917632s) [noise=true , address=145.79.227.5:4223 , piece_id=F25FX65PYUD43AYIG5JDT7APOMICUKKZHBWAF4X2RNFDDELRB3PQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.869702831 (596h42m19.740785876s) [noise=true , address=45.91.116.11:4120 , piece_id=DLCVHK7BZ4WYART4FDNPNAQOPN45NR3UA3WJNUUTFQIDMLKUL5OQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.870682406 (596h42m19.741765451s) [noise=true , address=77.90.31.19:4844 , piece_id=6SRRWZZKREJX5OPAYWOIA2WQDEKU2IIWS4I7RD47BFQRCPVABRDQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.870951348 (596h42m19.742034393s) [noise=true , address=45.91.116.6:4974 , piece_id=U3MW7UK4PL5NLWDDEO2BNOR2ACQHIICJBSTDKJFEQGFPQBK6H47A, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.871100612 (596h42m19.742183647s) [noise=true , address=145.79.227.20:4437 , piece_id=6WZ5ZBFFHBM5FYYGUSC5MCSRQPVKVPNJL4R4CZSOCSTGLCZ67UIA, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.871542814 (596h42m19.742625859s) [noise=true , address=45.91.116.12:4226 , piece_id=BUSX6WY6TEPFJCCUU7JSDRDQU577343GAYRPI4WY7Y4HK3RJA6IQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.871761761 (596h42m19.742844806s) [noise=true , address=145.79.227.12:4765 , piece_id=DK3SGCD4SHIGVBYG55MXZZTOQDQQC75TYI544AFMY2HHPF5JMVLQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.872053507 (596h42m19.743136562s) [noise=true , address=145.79.227.50:4373 , piece_id=LQWOR44DCAPMQUAWRJO2ADJYSPJVVWIZLRD7C7ZZE65X6BU6OK5A, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.872500619 (596h42m19.743583654s) [noise=true , address=145.79.227.21:4475 , piece_id=7FHO33R3CPUSP5ZIZRJHOURBJ2IM2GAPHXW454MMWPXEZP7HXVYQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.872988858 (596h42m19.744071893s) [noise=true , address=145.79.227.49:4337 , piece_id=64DWCETTCXREKVE3JCN3DATWWLMH3YXZ2UAJKSY7IAIFHPXMOSZA, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.873188868 (596h42m19.744271914s) [noise=true , address=45.91.116.5:4299 , piece_id=H2S5QH7RV76Q3WZC75ZAOSQVGVBNQGHJZSRCIBUNNXYMGN7SOTQA, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.873721814 (596h42m19.744804859s) [noise=true , address=145.79.227.47:4267 , piece_id=VZC3MBDIXSG4PNNZBBQNEKMGXVWYZKH5O6CLO3LJTMNYC35M5UGA, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.874269727 (596h42m19.745352772s) [noise=true , address=145.79.227.17:4972 , piece_id=WOHIEFST4MA6YCA5NADETCQJQE7PUFZJQSG3ZPE5AS7NFXT5U7FQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.874460500 (596h42m19.745543535s) [noise=true , address=145.79.227.3:4150 , piece_id=ZMCANIJDMBJ2N2GXTOBFVHWWVCIOWCEUSCAB7X2JAFZCUECJMR3Q, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:21.573995793 (596h42m21.445078838s) [upload=523063]: Next returned nil segment Dec 30 12:11:21.574013276 (596h42m21.445096321s) [upload=523063]: Waiting for error group managing segments... Dec 30 12:11:21.863634731 (596h42m21.734717776s) [noise=true , address=145.79.227.12:4765 , piece_id=DK3SGCD4SHIGVBYG55MXZZTOQDQQC75TYI544AFMY2HHPF5JMVLQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.864246045 (596h42m21.735329091s) [noise=true , address=145.79.227.50:4373 , piece_id=LQWOR44DCAPMQUAWRJO2ADJYSPJVVWIZLRD7C7ZZE65X6BU6OK5A, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.866264930 (596h42m21.737347975s) [noise=true , address=145.79.227.21:4475 , piece_id=7FHO33R3CPUSP5ZIZRJHOURBJ2IM2GAPHXW454MMWPXEZP7HXVYQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.866713603 (596h42m21.737796638s) [noise=true , address=145.79.227.20:4437 , piece_id=6WZ5ZBFFHBM5FYYGUSC5MCSRQPVKVPNJL4R4CZSOCSTGLCZ67UIA, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.866736467 (596h42m21.737819512s) [noise=true , address=145.79.227.47:4267 , piece_id=VZC3MBDIXSG4PNNZBBQNEKMGXVWYZKH5O6CLO3LJTMNYC35M5UGA, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.867367218 (596h42m21.738450253s) [noise=true , address=145.79.227.42:4123 , piece_id=6M4U2HP4OI63YLS5K243WSEPUUJXDH6UQ3IKANH4WZZADSDPFKPQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.883326547 (596h42m21.754409582s) [noise=true , address=145.79.227.41:4087 , piece_id=FIPQYMLA6VDUNYASNOTPYM2ABXLDQYC7UFCR2GV7YLQ6D7HDVK6Q, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.883959803 (596h42m21.755042838s) [noise=true , address=145.79.227.5:4223 , piece_id=F25FX65PYUD43AYIG5JDT7APOMICUKKZHBWAF4X2RNFDDELRB3PQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.942397537 (596h42m21.813480582s) [noise=true , address=145.79.227.3:4150 , piece_id=ZMCANIJDMBJ2N2GXTOBFVHWWVCIOWCEUSCAB7X2JAFZCUECJMR3Q, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.954529539 (596h42m21.825612584s) [noise=true , address=145.79.227.17:4972 , piece_id=WOHIEFST4MA6YCA5NADETCQJQE7PUFZJQSG3ZPE5AS7NFXT5U7FQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:22.048012075 (596h42m21.91909513s) [noise=true , address=77.90.31.19:4844 , piece_id=6SRRWZZKREJX5OPAYWOIA2WQDEKU2IIWS4I7RD47BFQRCPVABRDQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:22.173824922 (596h42m22.044907967s) [noise=true , address=45.91.116.12:4226 , piece_id=BUSX6WY6TEPFJCCUU7JSDRDQU577343GAYRPI4WY7Y4HK3RJA6IQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:22.199165705 (596h42m22.07024875s) [noise=true , address=45.91.116.6:4974 , piece_id=U3MW7UK4PL5NLWDDEO2BNOR2ACQHIICJBSTDKJFEQGFPQBK6H47A, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:22.199185723 (596h42m22.070268768s) [seg_pos={0 0}, upload=523063]: Segment reached optimal threshold of 13 pieces. Dec 30 12:11:22.199347832 (596h42m22.070430867s) [noise=true , address=45.91.116.5:4299 , piece_id=H2S5QH7RV76Q3WZC75ZAOSQVGVBNQGHJZSRCIBUNNXYMGN7SOTQA, seg_pos={0 0}, upload=523063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:43390->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:43390->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43390->45.91.116.5:4299: use of closed network connection Dec 30 12:11:22.199354344 (596h42m22.070437379s) [noise=true , address=45.91.116.5:4299 , piece_id=H2S5QH7RV76Q3WZC75ZAOSQVGVBNQGHJZSRCIBUNNXYMGN7SOTQA, seg_pos={0 0}, upload=523063]: Long tail cancellation due to optimal threshold Dec 30 12:11:22.199393749 (596h42m22.070476784s) [noise=true , address=145.79.227.49:4337 , piece_id=64DWCETTCXREKVE3JCN3DATWWLMH3YXZ2UAJKSY7IAIFHPXMOSZA, seg_pos={0 0}, upload=523063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 12:11:22.199402555 (596h42m22.07048559s) [noise=true , address=145.79.227.49:4337 , piece_id=64DWCETTCXREKVE3JCN3DATWWLMH3YXZ2UAJKSY7IAIFHPXMOSZA, seg_pos={0 0}, upload=523063]: Long tail cancellation due to optimal threshold Dec 30 12:11:22.199580625 (596h42m22.07066366s) [noise=true , address=45.91.116.11:4120 , piece_id=DLCVHK7BZ4WYART4FDNPNAQOPN45NR3UA3WJNUUTFQIDMLKUL5OQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:53024->45.91.116.11:4120: use of closed network connection; context canceled Dec 30 12:11:22.199587538 (596h42m22.070670573s) [noise=true , address=45.91.116.11:4120 , piece_id=DLCVHK7BZ4WYART4FDNPNAQOPN45NR3UA3WJNUUTFQIDMLKUL5OQ, seg_pos={0 0}, upload=523063]: Long tail cancellation due to optimal threshold Dec 30 12:11:22.199602877 (596h42m22.070685912s) [seg_pos={0 0}, upload=523063]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:11:22.199668783 (596h42m22.070751818s) [upload=523063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:11:22.199673742 (596h42m22.070756787s) [upload=523063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:11:22.486666723 (596h42m22.357749758s) [upload=523063]: Done uploading segments... Dec 30 12:11:22.486687102 (596h42m22.357770137s) [upload=523063]: Upload finished. err: <nil> Duration: 30.00372168s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....] duration: 3.839612075s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:11:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.839643746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.X........X................X.................X.........] duration: 42.112895504s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:11:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 42.112913839s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [....X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X....] duration: 3.846516639s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:10:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.846540526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X....X.X........X................X.................X........] duration: 38.613171254s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:10:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 38.613188067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.....X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X...] duration: 3.875546672s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:10:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.875599421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX....XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.] duration: 5.783742176s trace: https://jaeger.dev.storj.io/trace/492a9e8d5fc7dc42 FAIL (2025-12-30 12:10:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : false Dec 30 12:08:57.212307383 (596h39m57.083390418s) [upload=523027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:08:57.212311741 (596h39m57.083394776s) [upload=523027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:08:57.459667234 (596h39m57.330750269s) [seg_pos={0 0}, upload=523027]: Begin upload segment... Dec 30 12:08:57.459715666 (596h39m57.330798712s) [seg_pos={0 0}, upload=523027]: Done begin upload segment. Dec 30 12:08:57.459748138 (596h39m57.330831183s) [noise=true , address=145.79.227.52:4447 , piece_id=GYYS53W3I5AWFS4RZYKXJAJLYZOYNN7EIIFRE7EZ7RULGM4F2RNA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.459807671 (596h39m57.330890716s) [noise=true , address=145.79.227.47:4265 , piece_id=P7ANZHYVEGUK4XPD6YOPUU3Z6O3QE452QVK7XDFNMIPTBRF7ANNA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.460413305 (596h39m57.33149635s) [noise=true , address=145.79.227.10:4403 , piece_id=HGPYJZ5HLYREYHURJTCBPYAW7JAA3BV3PFK2L4SFZFY2NXLE74DQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.460922604 (596h39m57.332005639s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.461380026 (596h39m57.332463071s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462073897 (596h39m57.333156932s) [noise=true , address=77.90.31.21:4880 , piece_id=DRQJKEZZWZ5RLKK6IHI2H36ZQ25OJZFCNNHXPFGURVURS4IUTSZA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462176463 (596h39m57.333259498s) [noise=true , address=145.79.227.20:4437 , piece_id=AQBFBLAKB57YLPTDOBU4Q35OV3JO4DV77BMLOLBPHIABGKLEFCLA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462578488 (596h39m57.333661523s) [noise=true , address=45.91.116.12:4226 , piece_id=M2NGELGJGXIDYKI34WPG7UTDBGXSWK2533PN6IMEAAYF7S223XDA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462788808 (596h39m57.333871853s) [noise=true , address=45.91.116.7:4009 , piece_id=RY3RS3MIH6SB7IIKAB2EHZ535UIEMA7V4G5ZGGYLHK5YFUYW6TLA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462911692 (596h39m57.333994728s) [noise=true , address=145.79.227.11:4729 , piece_id=2QANA7EBWZ373DJ4UI6WDAEBKU7JFYVRYD2NRD6C6QEVUVE5UCCA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463286325 (596h39m57.33436937s) [noise=true , address=145.79.227.13:4803 , piece_id=2S3MUBR6BQJEGHHS7DUR2W3LGZ6EPVDFNBGUJYY2KW6QW2VEXHGQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463603229 (596h39m57.334686274s) [noise=true , address=45.91.116.5:4298 , piece_id=R4XKYCY366ROEMTQCUKPZZBW2DFWLRKYGPRRCAL5PRHR7NYEZQKA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463726333 (596h39m57.334809368s) [noise=true , address=145.79.227.7:4293 , piece_id=4YK3OS353A3UFCWTJE7DDOO3CEG5BUEC4CUUTEAJIQGHNSV3RPRA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464116827 (596h39m57.335199862s) [noise=true , address=145.79.227.45:4230 , piece_id=QX3NG7MHOZXWMWVXUKMHTG3ECG7HKVAKPW772OYD3V5NJ7XYU2IA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464391289 (596h39m57.335474334s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464453488 (596h39m57.335536533s) [noise=true , address=45.91.116.9:4264 , piece_id=5NPJU7J7BLWIMUFYN6ZXS73EZP6GL6RLERF6VK37L3542IBZALLQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:59.155709797 (596h39m59.026792842s) [upload=523027]: Next returned nil segment Dec 30 12:08:59.155730415 (596h39m59.02681346s) [upload=523027]: Waiting for error group managing segments... Dec 30 12:08:59.436845183 (596h39m59.307928218s) [noise=true , address=145.79.227.10:4403 , piece_id=HGPYJZ5HLYREYHURJTCBPYAW7JAA3BV3PFK2L4SFZFY2NXLE74DQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.439240203 (596h39m59.310323248s) [noise=true , address=145.79.227.45:4230 , piece_id=QX3NG7MHOZXWMWVXUKMHTG3ECG7HKVAKPW772OYD3V5NJ7XYU2IA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.443861913 (596h39m59.314944948s) [noise=true , address=145.79.227.11:4729 , piece_id=2QANA7EBWZ373DJ4UI6WDAEBKU7JFYVRYD2NRD6C6QEVUVE5UCCA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.451809968 (596h39m59.322893023s) [noise=true , address=145.79.227.20:4437 , piece_id=AQBFBLAKB57YLPTDOBU4Q35OV3JO4DV77BMLOLBPHIABGKLEFCLA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.467587110 (596h39m59.338670155s) [noise=true , address=145.79.227.13:4803 , piece_id=2S3MUBR6BQJEGHHS7DUR2W3LGZ6EPVDFNBGUJYY2KW6QW2VEXHGQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.480943984 (596h39m59.352027029s) [noise=true , address=145.79.227.47:4265 , piece_id=P7ANZHYVEGUK4XPD6YOPUU3Z6O3QE452QVK7XDFNMIPTBRF7ANNA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.537074725 (596h39m59.40815777s) [noise=true , address=145.79.227.52:4447 , piece_id=GYYS53W3I5AWFS4RZYKXJAJLYZOYNN7EIIFRE7EZ7RULGM4F2RNA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.770500565 (596h39m59.6415836s) [noise=true , address=145.79.227.7:4293 , piece_id=4YK3OS353A3UFCWTJE7DDOO3CEG5BUEC4CUUTEAJIQGHNSV3RPRA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.857879809 (596h39m59.728962844s) [noise=true , address=45.91.116.9:4264 , piece_id=5NPJU7J7BLWIMUFYN6ZXS73EZP6GL6RLERF6VK37L3542IBZALLQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.903950204 (596h39m59.775033249s) [noise=true , address=77.90.31.21:4880 , piece_id=DRQJKEZZWZ5RLKK6IHI2H36ZQ25OJZFCNNHXPFGURVURS4IUTSZA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.935948845 (596h39m59.80703189s) [noise=true , address=45.91.116.12:4226 , piece_id=M2NGELGJGXIDYKI34WPG7UTDBGXSWK2533PN6IMEAAYF7S223XDA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.955667292 (596h39m59.826750337s) [noise=true , address=45.91.116.5:4298 , piece_id=R4XKYCY366ROEMTQCUKPZZBW2DFWLRKYGPRRCAL5PRHR7NYEZQKA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:09:00.019740386 (596h39m59.890823421s) [noise=true , address=45.91.116.7:4009 , piece_id=RY3RS3MIH6SB7IIKAB2EHZ535UIEMA7V4G5ZGGYLHK5YFUYW6TLA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:09:00.019751929 (596h39m59.890834964s) [seg_pos={0 0}, upload=523027]: Segment reached optimal threshold of 13 pieces. Dec 30 12:09:00.019910320 (596h39m59.890993365s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection Dec 30 12:09:00.019918887 (596h39m59.891001922s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020351661 (596h39m59.891434706s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection Dec 30 12:09:00.020360327 (596h39m59.891443362s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020621204 (596h39m59.891704249s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:54168->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:09:00.020630592 (596h39m59.891713637s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020650039 (596h39m59.891733084s) [seg_pos={0 0}, upload=523027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:09:00.020713369 (596h39m59.891796414s) [upload=523027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:09:00.020718800 (596h39m59.891801835s) [upload=523027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:09:00.308291754 (596h40m0.179374789s) [upload=523027]: Done uploading segments... Dec 30 12:09:00.308310119 (596h40m0.179393164s) [upload=523027]: Upload finished. err: <nil> Duration: 5.783784056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X....X.X........X................X.................X.......] duration: 32.64362367s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:10:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 32.643655942s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X..] duration: 3.547215526s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:09:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547261785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX......XX] duration: 30.002808053s trace: https://jaeger.dev.storj.io/trace/492a9e8d5fc7dc42 FAIL (2025-12-30 12:09:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : false Dec 30 12:08:57.212307383 (596h39m57.083390418s) [upload=523027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:08:57.212311741 (596h39m57.083394776s) [upload=523027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:08:57.459667234 (596h39m57.330750269s) [seg_pos={0 0}, upload=523027]: Begin upload segment... Dec 30 12:08:57.459715666 (596h39m57.330798712s) [seg_pos={0 0}, upload=523027]: Done begin upload segment. Dec 30 12:08:57.459748138 (596h39m57.330831183s) [noise=true , address=145.79.227.52:4447 , piece_id=GYYS53W3I5AWFS4RZYKXJAJLYZOYNN7EIIFRE7EZ7RULGM4F2RNA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.459807671 (596h39m57.330890716s) [noise=true , address=145.79.227.47:4265 , piece_id=P7ANZHYVEGUK4XPD6YOPUU3Z6O3QE452QVK7XDFNMIPTBRF7ANNA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.460413305 (596h39m57.33149635s) [noise=true , address=145.79.227.10:4403 , piece_id=HGPYJZ5HLYREYHURJTCBPYAW7JAA3BV3PFK2L4SFZFY2NXLE74DQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.460922604 (596h39m57.332005639s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.461380026 (596h39m57.332463071s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462073897 (596h39m57.333156932s) [noise=true , address=77.90.31.21:4880 , piece_id=DRQJKEZZWZ5RLKK6IHI2H36ZQ25OJZFCNNHXPFGURVURS4IUTSZA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462176463 (596h39m57.333259498s) [noise=true , address=145.79.227.20:4437 , piece_id=AQBFBLAKB57YLPTDOBU4Q35OV3JO4DV77BMLOLBPHIABGKLEFCLA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462578488 (596h39m57.333661523s) [noise=true , address=45.91.116.12:4226 , piece_id=M2NGELGJGXIDYKI34WPG7UTDBGXSWK2533PN6IMEAAYF7S223XDA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462788808 (596h39m57.333871853s) [noise=true , address=45.91.116.7:4009 , piece_id=RY3RS3MIH6SB7IIKAB2EHZ535UIEMA7V4G5ZGGYLHK5YFUYW6TLA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462911692 (596h39m57.333994728s) [noise=true , address=145.79.227.11:4729 , piece_id=2QANA7EBWZ373DJ4UI6WDAEBKU7JFYVRYD2NRD6C6QEVUVE5UCCA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463286325 (596h39m57.33436937s) [noise=true , address=145.79.227.13:4803 , piece_id=2S3MUBR6BQJEGHHS7DUR2W3LGZ6EPVDFNBGUJYY2KW6QW2VEXHGQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463603229 (596h39m57.334686274s) [noise=true , address=45.91.116.5:4298 , piece_id=R4XKYCY366ROEMTQCUKPZZBW2DFWLRKYGPRRCAL5PRHR7NYEZQKA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463726333 (596h39m57.334809368s) [noise=true , address=145.79.227.7:4293 , piece_id=4YK3OS353A3UFCWTJE7DDOO3CEG5BUEC4CUUTEAJIQGHNSV3RPRA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464116827 (596h39m57.335199862s) [noise=true , address=145.79.227.45:4230 , piece_id=QX3NG7MHOZXWMWVXUKMHTG3ECG7HKVAKPW772OYD3V5NJ7XYU2IA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464391289 (596h39m57.335474334s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464453488 (596h39m57.335536533s) [noise=true , address=45.91.116.9:4264 , piece_id=5NPJU7J7BLWIMUFYN6ZXS73EZP6GL6RLERF6VK37L3542IBZALLQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:59.155709797 (596h39m59.026792842s) [upload=523027]: Next returned nil segment Dec 30 12:08:59.155730415 (596h39m59.02681346s) [upload=523027]: Waiting for error group managing segments... Dec 30 12:08:59.436845183 (596h39m59.307928218s) [noise=true , address=145.79.227.10:4403 , piece_id=HGPYJZ5HLYREYHURJTCBPYAW7JAA3BV3PFK2L4SFZFY2NXLE74DQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.439240203 (596h39m59.310323248s) [noise=true , address=145.79.227.45:4230 , piece_id=QX3NG7MHOZXWMWVXUKMHTG3ECG7HKVAKPW772OYD3V5NJ7XYU2IA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.443861913 (596h39m59.314944948s) [noise=true , address=145.79.227.11:4729 , piece_id=2QANA7EBWZ373DJ4UI6WDAEBKU7JFYVRYD2NRD6C6QEVUVE5UCCA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.451809968 (596h39m59.322893023s) [noise=true , address=145.79.227.20:4437 , piece_id=AQBFBLAKB57YLPTDOBU4Q35OV3JO4DV77BMLOLBPHIABGKLEFCLA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.467587110 (596h39m59.338670155s) [noise=true , address=145.79.227.13:4803 , piece_id=2S3MUBR6BQJEGHHS7DUR2W3LGZ6EPVDFNBGUJYY2KW6QW2VEXHGQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.480943984 (596h39m59.352027029s) [noise=true , address=145.79.227.47:4265 , piece_id=P7ANZHYVEGUK4XPD6YOPUU3Z6O3QE452QVK7XDFNMIPTBRF7ANNA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.537074725 (596h39m59.40815777s) [noise=true , address=145.79.227.52:4447 , piece_id=GYYS53W3I5AWFS4RZYKXJAJLYZOYNN7EIIFRE7EZ7RULGM4F2RNA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.770500565 (596h39m59.6415836s) [noise=true , address=145.79.227.7:4293 , piece_id=4YK3OS353A3UFCWTJE7DDOO3CEG5BUEC4CUUTEAJIQGHNSV3RPRA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.857879809 (596h39m59.728962844s) [noise=true , address=45.91.116.9:4264 , piece_id=5NPJU7J7BLWIMUFYN6ZXS73EZP6GL6RLERF6VK37L3542IBZALLQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.903950204 (596h39m59.775033249s) [noise=true , address=77.90.31.21:4880 , piece_id=DRQJKEZZWZ5RLKK6IHI2H36ZQ25OJZFCNNHXPFGURVURS4IUTSZA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.935948845 (596h39m59.80703189s) [noise=true , address=45.91.116.12:4226 , piece_id=M2NGELGJGXIDYKI34WPG7UTDBGXSWK2533PN6IMEAAYF7S223XDA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.955667292 (596h39m59.826750337s) [noise=true , address=45.91.116.5:4298 , piece_id=R4XKYCY366ROEMTQCUKPZZBW2DFWLRKYGPRRCAL5PRHR7NYEZQKA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:09:00.019740386 (596h39m59.890823421s) [noise=true , address=45.91.116.7:4009 , piece_id=RY3RS3MIH6SB7IIKAB2EHZ535UIEMA7V4G5ZGGYLHK5YFUYW6TLA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:09:00.019751929 (596h39m59.890834964s) [seg_pos={0 0}, upload=523027]: Segment reached optimal threshold of 13 pieces. Dec 30 12:09:00.019910320 (596h39m59.890993365s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection Dec 30 12:09:00.019918887 (596h39m59.891001922s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020351661 (596h39m59.891434706s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection Dec 30 12:09:00.020360327 (596h39m59.891443362s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020621204 (596h39m59.891704249s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:54168->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:09:00.020630592 (596h39m59.891713637s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020650039 (596h39m59.891733084s) [seg_pos={0 0}, upload=523027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:09:00.020713369 (596h39m59.891796414s) [upload=523027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:09:00.020718800 (596h39m59.891801835s) [upload=523027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:09:00.308291754 (596h40m0.179374789s) [upload=523027]: Done uploading segments... Dec 30 12:09:00.308310119 (596h40m0.179393164s) [upload=523027]: Upload finished. err: <nil> Duration: 30.002830275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X....X.X........X................X.................X......] duration: 26.393317696s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:09:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 26.393347854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.] duration: 3.725351224s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:08:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.725374418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X....X.X........X................X.................X.....] duration: 53.405820933s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:08:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 53.405841862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X] duration: 5.000848509s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:08:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000873246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX......X] duration: 30.00246224s trace: https://jaeger.dev.storj.io/trace/5d7bcf9f3c5a1a64 FAIL (2025-12-30 12:08:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: YTLXHLPTT2KHVBP7U7FXXVFQ3BKUKC64O6GZRIDNMGOBWNY7E2FA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: R243LP647T6KL5LJZDTAMTRIEFLOCW2ZOVH6KA5FEH7YFE6HK5EQ) context deadline exceeded ======LOGS====== Dec 30 12:08:03.131314867 (596h39m3.002397912s) [upload=523017]: Starting upload Dec 30 12:08:03.131419396 (596h39m3.002502441s) [upload=523017]: Done starting upload Dec 30 12:08:03.131513365 (596h39m3.00259641s) [upload=523017]: Uploading segments... Dec 30 12:08:03.131661838 (596h39m3.002744883s) [upload=523017]: Got next segment. Inline: false Dec 30 12:08:03.131702776 (596h39m3.002785831s) [upload=523017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:08:03.131709298 (596h39m3.002792343s) [upload=523017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:08:03.387728950 (596h39m3.258811985s) [seg_pos={0 0}, upload=523017]: Begin upload segment... Dec 30 12:08:03.387789816 (596h39m3.258872861s) [seg_pos={0 0}, upload=523017]: Done begin upload segment. Dec 30 12:08:03.387833669 (596h39m3.258916714s) [noise=true , address=145.79.227.35:4483 , piece_id=FNSBVUCMAVRTIDBJRXG3WAI4VGFBOMSH4RFVFWTDLCLO2KYPAIGA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.387851844 (596h39m3.258934889s) [noise=true , address=77.90.31.19:4842 , piece_id=YTLXHLPTT2KHVBP7U7FXXVFQ3BKUKC64O6GZRIDNMGOBWNY7E2FA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.388429764 (596h39m3.259512809s) [noise=true , address=145.79.227.52:4447 , piece_id=SRICKXI72QN2YMKVWB2ROKYD63BAZ5L2SUOLADIXMM3ZW7Z2MDVA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.388789320 (596h39m3.259872365s) [noise=true , address=77.90.31.10:4519 , piece_id=IPHI4CLPOAZ5BCDBPBKLLVQAUB65VFIFKLAKE7RMH6GF3ERSTA7A, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.389013396 (596h39m3.260096441s) [noise=true , address=145.79.227.17:4971 , piece_id=6KSLAVH2E33RULN67EOKYZUWMYXYNQTIBOUSJA6CDRMAYA2E4LWA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.389194500 (596h39m3.260277545s) [noise=true , address=145.79.227.16:4912 , piece_id=NX47O5R5HJ5BEQ4S6NUAG23AIYVP2R64NYCT4QB7RGWLKC7TDSVA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.389583601 (596h39m3.260666636s) [noise=true , address=45.91.116.10:4083 , piece_id=R243LP647T6KL5LJZDTAMTRIEFLOCW2ZOVH6KA5FEH7YFE6HK5EQ, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.389622286 (596h39m3.260705331s) [noise=true , address=77.90.31.21:4878 , piece_id=I6ZYJXHDZAKEALHXFLCTWZXEKQ77O6LLY75JDZS6VUT2RHBR3AQQ, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390119342 (596h39m3.261202377s) [noise=true , address=145.79.227.21:4475 , piece_id=LXYVYX67A7U7ZQV2KT3RQTSWCOSA6J35E6YG52S4VTOKQM3POYXA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390334000 (596h39m3.261417036s) [noise=true , address=145.79.227.39:4013 , piece_id=VR4YGLG6Y2LHUC7Q3WZJV7MZ564NX3UCFFRMIYU742ACUC4SCBWA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390432168 (596h39m3.261515203s) [noise=true , address=145.79.227.6:4258 , piece_id=ONHJ7P45GK2NCP4JFRB2VOEGBO3JUQUSUBCEL3MPDN6NAVDB2N3Q, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390766844 (596h39m3.261849879s) [noise=true , address=145.79.227.10:4403 , piece_id=ABTDMPJ3GHABTVAAL4WW36L2M4I5SROZODYFG5YMLCJXHFC3ZMRQ, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390872896 (596h39m3.261955941s) [noise=true , address=145.79.227.23:4545 , piece_id=QKHYQYE332CEBVFZA4W7RKRDJ6JDWPLTAANIYZOCGDQJ3JOIT3HA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.391074681 (596h39m3.262157726s) [noise=true , address=145.79.227.36:4518 , piece_id=QUYOJR6VWPXPKMYZMTEXHF7K6QO2XYYFVHXKEDNIGBMZRRU3OBKQ, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.391429867 (596h39m3.262512912s) [noise=true , address=145.79.227.51:4409 , piece_id=ZCBHG3WRIZ5K243KRZZL4SEFV7LU4WJC32UOK3PXAI7XVNBZAQVA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.391983150 (596h39m3.263066195s) [noise=true , address=45.91.116.6:4974 , piece_id=V4LTRBNVA4ZOQNLIRVWTV3A7A3DEAMZIHZVXXM3XVFP4D7ACIEIA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:05.095873405 (596h39m4.96695644s) [upload=523017]: Next returned nil segment Dec 30 12:08:05.095887863 (596h39m4.966970899s) [upload=523017]: Waiting for error group managing segments... Dec 30 12:08:05.380624156 (596h39m5.251707191s) [noise=true , address=145.79.227.6:4258 , piece_id=ONHJ7P45GK2NCP4JFRB2VOEGBO3JUQUSUBCEL3MPDN6NAVDB2N3Q, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.382375320 (596h39m5.253458355s) [noise=true , address=145.79.227.39:4013 , piece_id=VR4YGLG6Y2LHUC7Q3WZJV7MZ564NX3UCFFRMIYU742ACUC4SCBWA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.391486329 (596h39m5.262569364s) [noise=true , address=145.79.227.52:4447 , piece_id=SRICKXI72QN2YMKVWB2ROKYD63BAZ5L2SUOLADIXMM3ZW7Z2MDVA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.394502492 (596h39m5.265585537s) [noise=true , address=145.79.227.36:4518 , piece_id=QUYOJR6VWPXPKMYZMTEXHF7K6QO2XYYFVHXKEDNIGBMZRRU3OBKQ, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.398363314 (596h39m5.269446349s) [noise=true , address=145.79.227.17:4971 , piece_id=6KSLAVH2E33RULN67EOKYZUWMYXYNQTIBOUSJA6CDRMAYA2E4LWA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.400584834 (596h39m5.271667879s) [noise=true , address=145.79.227.10:4403 , piece_id=ABTDMPJ3GHABTVAAL4WW36L2M4I5SROZODYFG5YMLCJXHFC3ZMRQ, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.401015754 (596h39m5.272098799s) [noise=true , address=145.79.227.51:4409 , piece_id=ZCBHG3WRIZ5K243KRZZL4SEFV7LU4WJC32UOK3PXAI7XVNBZAQVA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.406963680 (596h39m5.278046725s) [noise=true , address=145.79.227.23:4545 , piece_id=QKHYQYE332CEBVFZA4W7RKRDJ6JDWPLTAANIYZOCGDQJ3JOIT3HA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.408678004 (596h39m5.27976105s) [noise=true , address=145.79.227.21:4475 , piece_id=LXYVYX67A7U7ZQV2KT3RQTSWCOSA6J35E6YG52S4VTOKQM3POYXA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.423604858 (596h39m5.294687903s) [noise=true , address=145.79.227.16:4912 , piece_id=NX47O5R5HJ5BEQ4S6NUAG23AIYVP2R64NYCT4QB7RGWLKC7TDSVA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.552469587 (596h39m5.423552632s) [noise=true , address=77.90.31.19:4842 , piece_id=YTLXHLPTT2KHVBP7U7FXXVFQ3BKUKC64O6GZRIDNMGOBWNY7E2FA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.675134440 (596h39m5.546217485s) [noise=true , address=77.90.31.10:4519 , piece_id=IPHI4CLPOAZ5BCDBPBKLLVQAUB65VFIFKLAKE7RMH6GF3ERSTA7A, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.791038610 (596h39m5.662121655s) [noise=true , address=45.91.116.10:4083 , piece_id=R243LP647T6KL5LJZDTAMTRIEFLOCW2ZOVH6KA5FEH7YFE6HK5EQ, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.791058307 (596h39m5.662141342s) [seg_pos={0 0}, upload=523017]: Segment reached optimal threshold of 13 pieces. Dec 30 12:08:05.791238139 (596h39m5.662321184s) [noise=true , address=145.79.227.35:4483 , piece_id=FNSBVUCMAVRTIDBJRXG3WAI4VGFBOMSH4RFVFWTDLCLO2KYPAIGA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 12:08:05.791244412 (596h39m5.662327457s) [noise=true , address=145.79.227.35:4483 , piece_id=FNSBVUCMAVRTIDBJRXG3WAI4VGFBOMSH4RFVFWTDLCLO2KYPAIGA, seg_pos={0 0}, upload=523017]: Long tail cancellation due to optimal threshold Dec 30 12:08:05.791278928 (596h39m5.662361963s) [noise=true , address=45.91.116.6:4974 , piece_id=V4LTRBNVA4ZOQNLIRVWTV3A7A3DEAMZIHZVXXM3XVFP4D7ACIEIA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:46000->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:46000->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46000->45.91.116.6:4974: use of closed network connection Dec 30 12:08:05.791287404 (596h39m5.662370439s) [noise=true , address=45.91.116.6:4974 , piece_id=V4LTRBNVA4ZOQNLIRVWTV3A7A3DEAMZIHZVXXM3XVFP4D7ACIEIA, seg_pos={0 0}, upload=523017]: Long tail cancellation due to optimal threshold Dec 30 12:08:05.791424244 (596h39m5.662507289s) [noise=true , address=77.90.31.21:4878 , piece_id=I6ZYJXHDZAKEALHXFLCTWZXEKQ77O6LLY75JDZS6VUT2RHBR3AQQ, seg_pos={0 0}, upload=523017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 30 12:08:05.791430436 (596h39m5.662513481s) [noise=true , address=77.90.31.21:4878 , piece_id=I6ZYJXHDZAKEALHXFLCTWZXEKQ77O6LLY75JDZS6VUT2RHBR3AQQ, seg_pos={0 0}, upload=523017]: Long tail cancellation due to optimal threshold Dec 30 12:08:05.791444963 (596h39m5.662528008s) [seg_pos={0 0}, upload=523017]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:08:05.791508534 (596h39m5.662591579s) [upload=523017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:08:05.791513323 (596h39m5.662596358s) [upload=523017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:08:06.086708351 (596h39m5.957791396s) [upload=523017]: Done uploading segments... Dec 30 12:08:06.086730914 (596h39m5.957813959s) [upload=523017]: Upload finished. err: <nil> Duration: 30.00248849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X....X.X........X................X.................X....] duration: 23.157201568s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 23.157221115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX......] duration: 5.736542561s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:07:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 5.736567578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...] duration: 3.681825366s trace: https://jaeger.dev.storj.io/trace/7a477c4bbcbc4acd FAIL (2025-12-30 12:07:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.681854963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X....X.X........X................X.................X...] duration: 36.025568011s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:07:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 36.025585033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX.....] duration: 5.690081162s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:07:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 5.690110666s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X..] duration: 3.571350875s trace: https://jaeger.dev.storj.io/trace/7a477c4bbcbc4acd FAIL (2025-12-30 12:06:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571377836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X....X.X........X................X.................X..] duration: 40.515802939s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 40.51583462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX....] duration: 6.002802998s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:06:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 6.002834266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X.] duration: 3.508194033s trace: https://jaeger.dev.storj.io/trace/7a477c4bbcbc4acd FAIL (2025-12-30 12:05:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508222038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX...] duration: 26.32153685s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:05:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 26.321565597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X....X.X........X................X.................X.] duration: 25.615993294s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:05:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 25.616007071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X] duration: 5.001404681s trace: https://jaeger.dev.storj.io/trace/7a477c4bbcbc4acd FAIL (2025-12-30 12:05:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001431112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X....X.X........X................X.................X] duration: 1m0.00225104s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:05:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 1m0.002272911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX..] duration: 5.978989637s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:04:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 5.979022981s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...] duration: 4.551289791s trace: https://jaeger.dev.storj.io/trace/2695c9304f1ad7a3 FAIL (2025-12-30 12:04:07 UTC) orbiter warning: context deadline exceeded Duration: 4.551321902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X....X.X........X................X.................] duration: 36.54001423s trace: https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134 FAIL (2025-12-30 12:04:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 1m57.317198375s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments... Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824 , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964 , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces. Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments... Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil> Duration: 36.540043106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX.] duration: 6.039889161s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:03:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 6.039908737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X..] duration: 3.639502316s trace: https://jaeger.dev.storj.io/trace/2695c9304f1ad7a3 FAIL (2025-12-30 12:03:26 UTC) orbiter warning: context deadline exceeded Duration: 3.639526091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX] duration: 30.002033549s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:03:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 30.002055772s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X....X.X........X................X................] duration: 43.96487187s trace: https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134 FAIL (2025-12-30 12:03:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 1m57.317198375s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments... Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824 , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964 , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces. Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments... Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil> Duration: 43.964887039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X.] duration: 5.001022975s trace: https://jaeger.dev.storj.io/trace/2695c9304f1ad7a3 FAIL (2025-12-30 12:02:53 UTC) orbiter warning: context deadline exceeded Duration: 5.001063913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X....X.X........X................X...............] duration: 28.844314021s trace: https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134 FAIL (2025-12-30 12:02:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 1m57.317198375s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments... Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824 , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964 , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces. Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments... Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil> Duration: 28.844371101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......X] duration: 30.002179076s trace: https://jaeger.dev.storj.io/trace/6878f4c6c3ed7d60 FAIL (2025-12-30 12:02:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 7HHFFPBEVCXIAIKWIOQXVMZJYY7DD7CEDYNNLUVCHBRHWL4QYQVA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 2TGN6BDCCHRJHZXX6GLEO42PDHMMSLPQMWUPYDVWBQY7HQHPPDPA) context deadline exceeded ======LOGS====== 12:01:56.213205520 (596h32m56.084288555s) [upload=522926]: Starting upload Dec 30 12:01:56.213264282 (596h32m56.084347317s) [upload=522926]: Done starting upload Dec 30 12:01:56.213419347 (596h32m56.084502382s) [upload=522926]: Uploading segments... Dec 30 12:01:56.213594370 (596h32m56.084677415s) [upload=522926]: Got next segment. Inline: false Dec 30 12:01:56.213696855 (596h32m56.0847799s) [upload=522926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:01:56.213704740 (596h32m56.084787785s) [upload=522926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:01:56.472695640 (596h32m56.343778685s) [seg_pos={0 0}, upload=522926]: Begin upload segment... Dec 30 12:01:56.472753650 (596h32m56.343836695s) [seg_pos={0 0}, upload=522926]: Done begin upload segment. Dec 30 12:01:56.472774019 (596h32m56.343857064s) [noise=true , address=77.90.31.16:4735 , piece_id=OH6TCLPE25J3ANKJRXS4LOZXZIPMZIDZ6NSHWERVTSJJNPY5WM6Q, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.473944147 (596h32m56.345027192s) [noise=true , address=145.79.227.47:4266 , piece_id=VKHGTXTV736ONW6EC6ZS4QO7UNACCRWQGH7QP6XPNYDCFRTYW63A, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.474382722 (596h32m56.345465757s) [noise=true , address=145.79.227.19:4077 , piece_id=Q7WUX7VW7CAA4WAGSAVNYWUKYL2Y4TQUONRPTQ6WL5TI3RL2IXJA, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.474753158 (596h32m56.345836193s) [noise=true , address=145.79.227.48:4302 , piece_id=VW6EUHONV7NV5UCCAT6ULW2CLZKBO457OKS2SGS43QMBQQ675HNA, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.475064832 (596h32m56.346147877s) [noise=true , address=45.91.116.10:4083 , piece_id=7HHFFPBEVCXIAIKWIOQXVMZJYY7DD7CEDYNNLUVCHBRHWL4QYQVA, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.475484270 (596h32m56.346567305s) [noise=true , address=45.91.116.13:4155 , piece_id=X2JS32WILYQAAQZRQSEY5EC44FEKVHWCMBDHAG53S43DMQYFAIKQ, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.476119239 (596h32m56.347202284s) [noise=true , address=145.79.227.35:4482 , piece_id=UOV4TLZGPOIQDG22H3GL67EZZILJAUCEPATWT7FHQEZTCBTIROCQ, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.476959108 (596h32m56.348042153s) [noise=true , address=145.79.227.2:4114 , piece_id=GSEML7HP5YIV5WBHBKR7Y56J6QEAP6PNVOINTRVT4ZJHJV7GS4CA, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.478408619 (596h32m56.349491694s) [noise=true , address=45.91.116.6:4976 , piece_id=2TGN6BDCCHRJHZXX6GLEO42PDHMMSLPQMWUPYDVWBQY7HQHPPDPA, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.482559312 (596h32m56.353642357s) [noise=true , address=145.79.227.46:4601 , piece_id=PUN7D2XKS2IJST42J2SHYZTOCL3XFXPNOW65MG2CR3CP27YE7B3Q, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.483021021 (596h32m56.354104056s) [noise=true , address=145.79.227.20:4437 , piece_id=YM33U6YXL5K3HOJ3H6KES5J5MP5FERKCLKX34TBMJC6RVD3P2YCA, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.483380786 (596h32m56.354463831s) [noise=true , address=45.91.116.11:4118 , piece_id=45BXMSXJXSVMY4YQ3ZF2TZVEOLALSNBVUXWFBW4H35YKQJ3JGPUA, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.483982522 (596h32m56.355065567s) [noise=true , address=145.79.227.29:4942 , piece_id=QRVWO7HBVRTGQC6FVXGV2N7ZWEQMNQO4DUL3LTSZJZJRBXUVXSAQ, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.485146839 (596h32m56.356229874s) [noise=true , address=145.79.227.45:4229 , piece_id=72H6BFMKYUFNFOW6AT3DI2SNOPDE7G2NBBLBZC2K7HFH66HZTD2A, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.485767871 (596h32m56.356850917s) [noise=true , address=145.79.227.26:4653 , piece_id=TVIXTTEJTEC2BNFUEJTLO4H4PVUN2SFFKX6MJ2WLDPQSNGTHDHAQ, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:56.486297229 (596h32m56.357380274s) [noise=true , address=145.79.227.12:4765 , piece_id=QBAKH3PMZIIVX7JMKZMR6BFFH2HXKM3L5L33LDIDJKWSXUFBUGPQ, seg_pos={0 0}, upload=522926]: Uploading piece... Dec 30 12:01:58.188360065 (596h32m58.0594431s) [upload=522926]: Next returned nil segment Dec 30 12:01:58.188400853 (596h32m58.059483908s) [upload=522926]: Waiting for error group managing segments... Dec 30 12:01:58.475499754 (596h32m58.346582789s) [noise=true , address=145.79.227.46:4601 , piece_id=PUN7D2XKS2IJST42J2SHYZTOCL3XFXPNOW65MG2CR3CP27YE7B3Q, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.477556361 (596h32m58.348639396s) [noise=true , address=145.79.227.45:4229 , piece_id=72H6BFMKYUFNFOW6AT3DI2SNOPDE7G2NBBLBZC2K7HFH66HZTD2A, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.479810673 (596h32m58.350893718s) [noise=true , address=145.79.227.12:4765 , piece_id=QBAKH3PMZIIVX7JMKZMR6BFFH2HXKM3L5L33LDIDJKWSXUFBUGPQ, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.484697339 (596h32m58.355780374s) [noise=true , address=145.79.227.2:4114 , piece_id=GSEML7HP5YIV5WBHBKR7Y56J6QEAP6PNVOINTRVT4ZJHJV7GS4CA, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.489376218 (596h32m58.360459253s) [noise=true , address=145.79.227.19:4077 , piece_id=Q7WUX7VW7CAA4WAGSAVNYWUKYL2Y4TQUONRPTQ6WL5TI3RL2IXJA, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.497399715 (596h32m58.36848275s) [noise=true , address=145.79.227.20:4437 , piece_id=YM33U6YXL5K3HOJ3H6KES5J5MP5FERKCLKX34TBMJC6RVD3P2YCA, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.500602504 (596h32m58.371685549s) [noise=true , address=145.79.227.29:4942 , piece_id=QRVWO7HBVRTGQC6FVXGV2N7ZWEQMNQO4DUL3LTSZJZJRBXUVXSAQ, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.503990596 (596h32m58.375073641s) [noise=true , address=145.79.227.47:4266 , piece_id=VKHGTXTV736ONW6EC6ZS4QO7UNACCRWQGH7QP6XPNYDCFRTYW63A, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.554271929 (596h32m58.425354974s) [noise=true , address=145.79.227.48:4302 , piece_id=VW6EUHONV7NV5UCCAT6ULW2CLZKBO457OKS2SGS43QMBQQ675HNA, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.575335477 (596h32m58.446418513s) [noise=true , address=145.79.227.26:4653 , piece_id=TVIXTTEJTEC2BNFUEJTLO4H4PVUN2SFFKX6MJ2WLDPQSNGTHDHAQ, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.809431613 (596h32m58.680514658s) [noise=true , address=45.91.116.10:4083 , piece_id=7HHFFPBEVCXIAIKWIOQXVMZJYY7DD7CEDYNNLUVCHBRHWL4QYQVA, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.820644224 (596h32m58.691727269s) [noise=true , address=45.91.116.11:4118 , piece_id=45BXMSXJXSVMY4YQ3ZF2TZVEOLALSNBVUXWFBW4H35YKQJ3JGPUA, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.995957689 (596h32m58.867040724s) [noise=true , address=45.91.116.6:4976 , piece_id=2TGN6BDCCHRJHZXX6GLEO42PDHMMSLPQMWUPYDVWBQY7HQHPPDPA, seg_pos={0 0}, upload=522926]: Done uploading piece. err: <nil> Dec 30 12:01:58.995975031 (596h32m58.867058066s) [seg_pos={0 0}, upload=522926]: Segment reached optimal threshold of 13 pieces. Dec 30 12:01:58.996194079 (596h32m58.867277114s) [noise=true , address=77.90.31.16:4735 , piece_id=OH6TCLPE25J3ANKJRXS4LOZXZIPMZIDZ6NSHWERVTSJJNPY5WM6Q, seg_pos={0 0}, upload=522926]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 12:01:58.996200441 (596h32m58.867283476s) [noise=true , address=77.90.31.16:4735 , piece_id=OH6TCLPE25J3ANKJRXS4LOZXZIPMZIDZ6NSHWERVTSJJNPY5WM6Q, seg_pos={0 0}, upload=522926]: Long tail cancellation due to optimal threshold Dec 30 12:01:58.996259332 (596h32m58.867342367s) [noise=true , address=45.91.116.13:4155 , piece_id=X2JS32WILYQAAQZRQSEY5EC44FEKVHWCMBDHAG53S43DMQYFAIKQ, seg_pos={0 0}, upload=522926]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:34902->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:34902->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->45.91.116.13:4155: use of closed network connection Dec 30 12:01:58.996306112 (596h32m58.867389157s) [noise=true , address=45.91.116.13:4155 , piece_id=X2JS32WILYQAAQZRQSEY5EC44FEKVHWCMBDHAG53S43DMQYFAIKQ, seg_pos={0 0}, upload=522926]: Long tail cancellation due to optimal threshold Dec 30 12:01:58.996389571 (596h32m58.867472606s) [noise=true , address=145.79.227.35:4482 , piece_id=UOV4TLZGPOIQDG22H3GL67EZZILJAUCEPATWT7FHQEZTCBTIROCQ, seg_pos={0 0}, upload=522926]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 12:01:58.996395522 (596h32m58.867478557s) [noise=true , address=145.79.227.35:4482 , piece_id=UOV4TLZGPOIQDG22H3GL67EZZILJAUCEPATWT7FHQEZTCBTIROCQ, seg_pos={0 0}, upload=522926]: Long tail cancellation due to optimal threshold Dec 30 12:01:58.996411653 (596h32m58.867494688s) [seg_pos={0 0}, upload=522926]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:01:58.996538203 (596h32m58.867621238s) [upload=522926]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:01:58.996544196 (596h32m58.867627241s) [upload=522926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:01:59.283469116 (596h32m59.154552161s) [upload=522926]: Done uploading segments... Dec 30 12:01:59.283487570 (596h32m59.154570605s) [upload=522926]: Upload finished. err: <nil> Duration: 30.002207911s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X] duration: 5.001046496s trace: https://jaeger.dev.storj.io/trace/2695c9304f1ad7a3 FAIL (2025-12-30 12:01:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001067507s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X...........X....X.X........X................X..............] duration: 35.046235422s trace: https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134 FAIL (2025-12-30 12:01:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 1m57.317198375s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments... Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824 , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964 , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces. Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments... Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil> Duration: 35.046266229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.] duration: 3.424096289s trace: https://jaeger.dev.storj.io/trace/7a50dbe067dd4237 FAIL (2025-12-30 12:01:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424121896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......] duration: 6.06878391s trace: https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79 FAIL (2025-12-30 12:01:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment... Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment. Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330 , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155 , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365 , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086 , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912 , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765 , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221 , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191 , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978 , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006 , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265 , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187 , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566 , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments... Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765 , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086 , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978 , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566 , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265 , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365 , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221 , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187 , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006 , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330 , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912 , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191 , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155 , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces. Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments... Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil> Duration:* 6.068828856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...........X....X.X........X................X.............] duration: 15.42755873s trace: https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134 FAIL (2025-12-30 12:00:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 1m57.317198375s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments... Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824 , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964 , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces. Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments... Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil> Duration: 15.427598996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX] duration: 5.001356849s trace: https://jaeger.dev.storj.io/trace/7a50dbe067dd4237 FAIL (2025-12-30 12:00:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00138332s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...........X....X.X........X................X............] duration: 15.566150849s trace: https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134 FAIL (2025-12-30 12:00:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 1m57.317198375s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments... Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824 , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964 , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces. Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments... Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil> Duration: 15.566192327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X......] duration: 5.972417522s trace: https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79 FAIL (2025-12-30 11:59:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment... Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment. Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330 , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155 , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365 , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086 , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912 , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765 , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221 , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191 , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978 , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006 , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265 , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187 , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566 , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments... Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765 , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086 , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978 , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566 , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265 , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365 , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221 , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187 , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006 , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330 , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912 , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191 , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155 , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces. Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments... Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil> Duration:* 5.972452449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...........X....X.X........X................X...........] duration: 40.89724796s trace: https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134 FAIL (2025-12-30 11:59:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 1m57.317198375s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments... Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824 , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964 , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces. Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments... Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil> Duration: 40.897275062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X.....] duration: 5.895534863s trace: https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79 FAIL (2025-12-30 11:59:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment... Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment. Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330 , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155 , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365 , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086 , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912 , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765 , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221 , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191 , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978 , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006 , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265 , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187 , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566 , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece... Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments... Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765 , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086 , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978 , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566 , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265 , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365 , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221 , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187 , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006 , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330 , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912 , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191 , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155 , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil> Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces. Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297 , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937 , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117 , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments... Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil> Duration:* 5.895557395s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXX] duration: 5.000882803s trace: https://jaeger.dev.storj.io/trace/70d07e2c957d00b5 FAIL (2025-12-30 11:59:33 UTC) orbiter warning: context deadline exceeded Duration: 5.000903964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXX] duration: 5.001385602s trace: https://jaeger.dev.storj.io/trace/45a919a26494f152 FAIL (2025-12-30 11:59:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001412964s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)