Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........X.XXX...X..X...................XX.X........X.XXX.X] duration: 1m0.00225286s trace: https://jaeger.dev.storj.io/trace/7836d9461cf629ea FAIL (2025-12-26 13:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: SVS2GDGXGSVOJVOD76QFJZUBJZK2AQ5BDERRQEHVMQD3CMZZIK4A) context deadline exceeded; piecestore: (Node ID: 12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, Piece ID: ALU4TSZJFFJJDWVRB3MSD6C54XKMXUWHVLZ2QKMQWPFZKSMGXPLA) context deadline exceeded; piecestore: (Node ID: 12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx, Piece ID: NU42OBHMH75VZBU5AH65YEBQWRURMVWEUYWIUEW6P5TVEMMRSN4A) context deadline exceeded; piecestore: (Node ID: 12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, Piece ID: YN6CC3SOETUO37XI3AYLRIPE65HPZMNOUHSV37FARVABIPF362AA) context deadline exceeded ======LOGS====== 13:4637 , piece_id=OYHTZ3JLEQX7T6Q42UY2KQWTL5UUGLJYYSRYIFYGH2EOWV6ANJHQ, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.796647271 (501h55m41.667730307s) [noise=true , address=77.90.31.21:4897 , piece_id=YN6CC3SOETUO37XI3AYLRIPE65HPZMNOUHSV37FARVABIPF362AA, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.796806965 (501h55m41.66789s) [noise=true , address=77.90.31.11:4587 , piece_id=SVS2GDGXGSVOJVOD76QFJZUBJZK2AQ5BDERRQEHVMQD3CMZZIK4A, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.796903138 (501h55m41.667986183s) [noise=true , address=145.79.227.18:4017 , piece_id=RXCHMPTZ3BDRZKBPDI33R2Z4D5MGFFX5OYZ4PFE6G6B7UZPBF7WQ, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797216675 (501h55m41.66829972s) [noise=true , address=145.79.227.50:4407 , piece_id=S6YTNV2G5WMCQLLXTHAQYY5EXDROLAHUED546ARBO6X2X773SJFA, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797546172 (501h55m41.668629217s) [noise=true , address=145.79.227.40:4077 , piece_id=VCULWJBKZWECS7RM3NEPNZ5VUB6DXQRPP6QTQS725ZIRTDUEQNFA, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797551623 (501h55m41.668634658s) [noise=true , address=145.79.227.21:4505 , piece_id=2BSYTCNQNSXIAVNX6GLABLMWYF6BIFWCGNDQDCJXVV6Z6OGUCJMA, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797554880 (501h55m41.668637915s) [noise=true , address=45.91.116.9:4286 , piece_id=QI53DQYSOQ35CWV4PTEER743MVNNK4I4QGL7X3ZMOU27EYIRWK4Q, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797557985 (501h55m41.668641021s) [noise=true , address=77.90.31.17:4774 , piece_id=ALU4TSZJFFJJDWVRB3MSD6C54XKMXUWHVLZ2QKMQWPFZKSMGXPLA, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797560881 (501h55m41.668643916s) [noise=true , address=77.90.31.19:4865 , piece_id=2UIUFVCWI4ZOCHI5X6LMPB4RWJAAVSF3YFSDW6FSTGXB3IYSSQ6A, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797563937 (501h55m41.668646972s) [noise=true , address=45.91.116.5:4302 , piece_id=NU42OBHMH75VZBU5AH65YEBQWRURMVWEUYWIUEW6P5TVEMMRSN4A, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797566592 (501h55m41.668649637s) [noise=true , address=77.90.31.18:4813 , piece_id=JC4KDCFIVPAFYP3MRA23WBDPXPERATHEIHGXPT6QTOMG6M7L2EMQ, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.797569457 (501h55m41.668652492s) [noise=true , address=64.49.13.169:4707 , piece_id=4D762HI62Y6S2IC246YG6PKZPUR2R3JD4KYC6FDAPU63Y4G6NI6Q, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.798140684 (501h55m41.669223719s) [noise=true , address=145.79.227.52:4469 , piece_id=NI6BF3C545G3JZ3SV36YSJQOUDCS26K7XYG4XDNAOLJP65LNPMWA, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:41.798507002 (501h55m41.669590037s) [noise=true , address=145.79.227.44:4228 , piece_id=DW7PSUIFVCJOCEYJ3BSDEZYZMIZPKJAOQBU64WWLUNBH3ZI7KNVA, seg_pos={0 0}, upload=440502]: Uploading piece... Dec 26 13:24:43.263377087 (501h55m43.134460122s) [upload=440502]: Next returned nil segment Dec 26 13:24:43.263399941 (501h55m43.134482976s) [upload=440502]: Waiting for error group managing segments... Dec 26 13:24:43.682130308 (501h55m43.553213353s) [noise=true , address=145.79.227.52:4469 , piece_id=NI6BF3C545G3JZ3SV36YSJQOUDCS26K7XYG4XDNAOLJP65LNPMWA, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:43.682979224 (501h55m43.554062269s) [noise=true , address=145.79.227.21:4505 , piece_id=2BSYTCNQNSXIAVNX6GLABLMWYF6BIFWCGNDQDCJXVV6Z6OGUCJMA, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:43.701305561 (501h55m43.572388606s) [noise=true , address=145.79.227.50:4407 , piece_id=S6YTNV2G5WMCQLLXTHAQYY5EXDROLAHUED546ARBO6X2X773SJFA, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:43.703549093 (501h55m43.574632128s) [noise=true , address=145.79.227.40:4077 , piece_id=VCULWJBKZWECS7RM3NEPNZ5VUB6DXQRPP6QTQS725ZIRTDUEQNFA, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:43.704162481 (501h55m43.575245526s) [noise=true , address=145.79.227.18:4017 , piece_id=RXCHMPTZ3BDRZKBPDI33R2Z4D5MGFFX5OYZ4PFE6G6B7UZPBF7WQ, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:43.718116642 (501h55m43.589199697s) [noise=true , address=145.79.227.43:4173 , piece_id=T6Y5J7CKAU2IO6IVOYBZGVAK3WEVWIVBT47OLNEG2EJGHOK7MIFQ, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:43.937384507 (501h55m43.808467552s) [noise=true , address=77.90.31.18:4813 , piece_id=JC4KDCFIVPAFYP3MRA23WBDPXPERATHEIHGXPT6QTOMG6M7L2EMQ, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:43.947589480 (501h55m43.818672525s) [noise=true , address=77.90.31.21:4897 , piece_id=YN6CC3SOETUO37XI3AYLRIPE65HPZMNOUHSV37FARVABIPF362AA, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:43.971663050 (501h55m43.842746085s) [noise=true , address=77.90.31.17:4774 , piece_id=ALU4TSZJFFJJDWVRB3MSD6C54XKMXUWHVLZ2QKMQWPFZKSMGXPLA, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:44.001149808 (501h55m43.872232853s) [noise=true , address=45.91.116.5:4302 , piece_id=NU42OBHMH75VZBU5AH65YEBQWRURMVWEUYWIUEW6P5TVEMMRSN4A, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:44.257968357 (501h55m44.129051402s) [noise=true , address=77.90.31.11:4587 , piece_id=SVS2GDGXGSVOJVOD76QFJZUBJZK2AQ5BDERRQEHVMQD3CMZZIK4A, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:44.310637488 (501h55m44.181720533s) [noise=true , address=77.90.31.19:4865 , piece_id=2UIUFVCWI4ZOCHI5X6LMPB4RWJAAVSF3YFSDW6FSTGXB3IYSSQ6A, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:44.560486786 (501h55m44.431569851s) [noise=true , address=145.79.227.30:4782 , piece_id=554ZLXI7AZ3R2YE6U6CK7RAT2SOIQ5ZAGHZVBMGKKOA75QAQUNWQ, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:44.664244991 (501h55m44.535328026s) [noise=true , address=145.79.227.34:4940 , piece_id=PPPFXYRRELPLLVBS2TEMK4DZ6CG5Y7CBSE66EY74EPQIVYEWU6OQ, seg_pos={0 0}, upload=440502]: Done uploading piece. err: <nil> Dec 26 13:24:44.664260510 (501h55m44.535343545s) [seg_pos={0 0}, upload=440502]: Segment reached optimal threshold of 14 pieces. Dec 26 13:24:44.664584617 (501h55m44.535667662s) [noise=true , address=64.49.13.169:4707 , piece_id=4D762HI62Y6S2IC246YG6PKZPUR2R3JD4KYC6FDAPU63Y4G6NI6Q, seg_pos={0 0}, upload=440502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:53148->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:53148->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53148->64.49.13.169:4707: use of closed network connection Dec 26 13:24:44.664591461 (501h55m44.535674496s) [noise=true , address=64.49.13.169:4707 , piece_id=4D762HI62Y6S2IC246YG6PKZPUR2R3JD4KYC6FDAPU63Y4G6NI6Q, seg_pos={0 0}, upload=440502]: Long tail cancellation due to optimal threshold Dec 26 13:24:44.664691791 (501h55m44.535774826s) [noise=true , address=145.79.227.44:4228 , piece_id=DW7PSUIFVCJOCEYJ3BSDEZYZMIZPKJAOQBU64WWLUNBH3ZI7KNVA, seg_pos={0 0}, upload=440502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X): protocol: expected piece hash; context canceled; EOF Dec 26 13:24:44.664696781 (501h55m44.535779816s) [noise=true , address=145.79.227.44:4228 , piece_id=DW7PSUIFVCJOCEYJ3BSDEZYZMIZPKJAOQBU64WWLUNBH3ZI7KNVA, seg_pos={0 0}, upload=440502]: Long tail cancellation due to optimal threshold Dec 26 13:24:44.665024475 (501h55m44.53610751s) [noise=true , address=45.91.116.9:4286 , piece_id=QI53DQYSOQ35CWV4PTEER743MVNNK4I4QGL7X3ZMOU27EYIRWK4Q, seg_pos={0 0}, upload=440502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa): protocol: expected piece hash; context canceled; EOF Dec 26 13:24:44.665030676 (501h55m44.536113711s) [noise=true , address=45.91.116.9:4286 , piece_id=QI53DQYSOQ35CWV4PTEER743MVNNK4I4QGL7X3ZMOU27EYIRWK4Q, seg_pos={0 0}, upload=440502]: Long tail cancellation due to optimal threshold Dec 26 13:24:44.665371445 (501h55m44.53645448s) [noise=true , address=77.90.31.13:4637 , piece_id=OYHTZ3JLEQX7T6Q42UY2KQWTL5UUGLJYYSRYIFYGH2EOWV6ANJHQ, seg_pos={0 0}, upload=440502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: write tcp 188.166.225.170:55434->77.90.31.13:4637: use of closed network connection; write tcp 188.166.225.170:55434->77.90.31.13:4637: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55434->77.90.31.13:4637: use of closed network connection Dec 26 13:24:44.665378348 (501h55m44.536461383s) [noise=true , address=77.90.31.13:4637 , piece_id=OYHTZ3JLEQX7T6Q42UY2KQWTL5UUGLJYYSRYIFYGH2EOWV6ANJHQ, seg_pos={0 0}, upload=440502]: Long tail cancellation due to optimal threshold Dec 26 13:24:44.665502515 (501h55m44.53658555s) [seg_pos={0 0}, upload=440502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:24:44.665596033 (501h55m44.536679068s) [upload=440502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:24:44.665601253 (501h55m44.536684288s) [upload=440502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:24:44.970528840 (501h55m44.841611875s) [upload=440502]: Done uploading segments... Dec 26 13:24:44.970547996 (501h55m44.841631042s) [upload=440502]: Upload finished. err: <nil> Duration: 1m0.002276927s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XX] duration: 5.001923325s trace: https://jaeger.dev.storj.io/trace/9393dc757ee8a7a FAIL (2025-12-26 13:25:32 UTC) orbiter warning: context deadline exceeded Duration: 5.001947672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X...] duration: 6.272654104s trace: https://jaeger.dev.storj.io/trace/70ecae77a4804cc FAIL (2025-12-26 13:25:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA) context deadline exceeded ======LOGS====== Dec 26 13:22:32.436256516 (501h53m32.307339561s) [upload=440470]: Starting upload Dec 26 13:22:32.436335557 (501h53m32.307418622s) [upload=440470]: Done starting upload Dec 26 13:22:32.436536680 (501h53m32.307619725s) [upload=440470]: Uploading segments... Dec 26 13:22:32.436601352 (501h53m32.307684407s) [upload=440470]: Got next segment. Inline: false Dec 26 13:22:32.436635207 (501h53m32.307718242s) [upload=440470]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:22:32.436641259 (501h53m32.307724304s) [upload=440470]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:22:32.680370322 (501h53m32.551453357s) [seg_pos={0 0}, upload=440470]: Begin upload segment... Dec 26 13:22:32.680425708 (501h53m32.551508743s) [seg_pos={0 0}, upload=440470]: Done begin upload segment. Dec 26 13:22:32.680452900 (501h53m32.551535935s) [noise=true , address=77.90.31.13:4627 , piece_id=SHXBCOH4RVCH3LTMDY6T2FOXQUY42MFFQL6QUR7N6LZL65PF56NA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.680490111 (501h53m32.551573176s) [noise=true , address=77.90.31.16:4735 , piece_id=ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.681369505 (501h53m32.55245254s) [noise=true , address=145.79.227.28:4727 , piece_id=SA3AYCPVMMTMD5GQKBNSRSVSDBN6APHPLRSVUTLIKA3YAJFCIJHA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.682075288 (501h53m32.553158323s) [noise=true , address=145.79.227.13:4802 , piece_id=PVACSPBULCNEYH3DED2VIIPVB4FPRBBEYVNBEH4VQWHH3BPNWGBA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.683095442 (501h53m32.554178477s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.684650502 (501h53m32.555733537s) [noise=true , address=145.79.227.19:4078 , piece_id=XIGHVBQOBXXURFLGSQC54WCV3PT6IS2VKNBX3IIMDNFF7UM2UT7Q, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.685456448 (501h53m32.556539483s) [noise=true , address=145.79.227.20:4438 , piece_id=ASLXFWRJUW4BHSX2PKZK4JIJJDRDXIRCAG2VXQCTUR62NP76GO7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.686754629 (501h53m32.557837664s) [noise=true , address=45.91.116.8:4045 , piece_id=TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.687518043 (501h53m32.558601078s) [noise=true , address=145.79.227.38:4978 , piece_id=JLOJJ5K6MLTKDKT2EBDZGWAY6HJQCID4UBNHZAOVNXELP5YVHU7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.693524029 (501h53m32.564607064s) [noise=true , address=145.79.227.45:4231 , piece_id=XZIGODMIDD5BQFXINLEUHZEON6BQANHSLIETFV55UC4NVMD7E5GA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694151704 (501h53m32.565234739s) [noise=true , address=145.79.227.37:4567 , piece_id=LTIMLA6I4ZJLYOTUGGROMJ7N3IAQ7XYLDFCUMJWPJ4UHPK32GLQQ, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694519495 (501h53m32.56560253s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694951247 (501h53m32.566034282s) [noise=true , address=145.79.227.39:4015 , piece_id=DI2J3QCAO55CGHNSEHVMXU46I4P77NZNGX5MW3I2DN6ISYJ4JT3A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.695331851 (501h53m32.566414896s) [noise=true , address=145.79.227.6:4258 , piece_id=HKLVNVOO3MIZ6KUNAHQGMRQSGFA37TKA54N34HAYRFRUGWPKCQ6Q, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.695741902 (501h53m32.566824948s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.696147946 (501h53m32.567230991s) [noise=true , address=45.91.116.6:4976 , piece_id=PYPVPYVEI6D22PVBBB37EMWUEXM3YKJ5BTIL7MAMXUAX5AL2XVPA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:34.406954778 (501h53m34.278037813s) [upload=440470]: Next returned nil segment Dec 26 13:22:34.406968063 (501h53m34.278051098s) [upload=440470]: Waiting for error group managing segments... Dec 26 13:22:34.696844389 (501h53m34.567927424s) [noise=true , address=145.79.227.28:4727 , piece_id=SA3AYCPVMMTMD5GQKBNSRSVSDBN6APHPLRSVUTLIKA3YAJFCIJHA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.704462957 (501h53m34.575546002s) [noise=true , address=145.79.227.38:4978 , piece_id=JLOJJ5K6MLTKDKT2EBDZGWAY6HJQCID4UBNHZAOVNXELP5YVHU7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.706884898 (501h53m34.577967943s) [noise=true , address=145.79.227.13:4802 , piece_id=PVACSPBULCNEYH3DED2VIIPVB4FPRBBEYVNBEH4VQWHH3BPNWGBA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.710527174 (501h53m34.581610219s) [noise=true , address=145.79.227.6:4258 , piece_id=HKLVNVOO3MIZ6KUNAHQGMRQSGFA37TKA54N34HAYRFRUGWPKCQ6Q, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.716355412 (501h53m34.587438487s) [noise=true , address=145.79.227.45:4231 , piece_id=XZIGODMIDD5BQFXINLEUHZEON6BQANHSLIETFV55UC4NVMD7E5GA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.722119418 (501h53m34.593202463s) [noise=true , address=145.79.227.20:4438 , piece_id=ASLXFWRJUW4BHSX2PKZK4JIJJDRDXIRCAG2VXQCTUR62NP76GO7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.722645529 (501h53m34.593728575s) [noise=true , address=145.79.227.19:4078 , piece_id=XIGHVBQOBXXURFLGSQC54WCV3PT6IS2VKNBX3IIMDNFF7UM2UT7Q, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.731926392 (501h53m34.603009437s) [noise=true , address=145.79.227.37:4567 , piece_id=LTIMLA6I4ZJLYOTUGGROMJ7N3IAQ7XYLDFCUMJWPJ4UHPK32GLQQ, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.741636291 (501h53m34.612719336s) [noise=true , address=145.79.227.39:4015 , piece_id=DI2J3QCAO55CGHNSEHVMXU46I4P77NZNGX5MW3I2DN6ISYJ4JT3A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.966402947 (501h53m34.837485992s) [noise=true , address=45.91.116.6:4976 , piece_id=PYPVPYVEI6D22PVBBB37EMWUEXM3YKJ5BTIL7MAMXUAX5AL2XVPA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.200199657 (501h53m35.071282702s) [noise=true , address=77.90.31.13:4627 , piece_id=SHXBCOH4RVCH3LTMDY6T2FOXQUY42MFFQL6QUR7N6LZL65PF56NA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.482129723 (501h53m35.353212758s) [noise=true , address=45.91.116.8:4045 , piece_id=TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.673121025 (501h53m35.54420406s) [noise=true , address=77.90.31.16:4735 , piece_id=ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.673140882 (501h53m35.544223917s) [seg_pos={0 0}, upload=440470]: Segment reached optimal threshold of 13 pieces. Dec 26 13:22:35.673324491 (501h53m35.544407536s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673332608 (501h53m35.544415653s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673511408 (501h53m35.544594443s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673517529 (501h53m35.544600574s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673559619 (501h53m35.544642654s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673580289 (501h53m35.544663334s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673627980 (501h53m35.544711025s) [seg_pos={0 0}, upload=440470]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:22:35.673776222 (501h53m35.544859277s) [upload=440470]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:22:35.673784368 (501h53m35.544867413s) [upload=440470]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:22:35.965152704 (501h53m35.836235749s) [upload=440470]: Done uploading segments... Dec 26 13:22:35.965174246 (501h53m35.836257291s) [upload=440470]: Upload finished. err: <nil> Duration: 6.272681377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X......X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X..] duration: 6.162319708s trace: https://jaeger.dev.storj.io/trace/70ecae77a4804cc FAIL (2025-12-26 13:24:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA) context deadline exceeded ======LOGS====== Dec 26 13:22:32.436256516 (501h53m32.307339561s) [upload=440470]: Starting upload Dec 26 13:22:32.436335557 (501h53m32.307418622s) [upload=440470]: Done starting upload Dec 26 13:22:32.436536680 (501h53m32.307619725s) [upload=440470]: Uploading segments... Dec 26 13:22:32.436601352 (501h53m32.307684407s) [upload=440470]: Got next segment. Inline: false Dec 26 13:22:32.436635207 (501h53m32.307718242s) [upload=440470]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:22:32.436641259 (501h53m32.307724304s) [upload=440470]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:22:32.680370322 (501h53m32.551453357s) [seg_pos={0 0}, upload=440470]: Begin upload segment... Dec 26 13:22:32.680425708 (501h53m32.551508743s) [seg_pos={0 0}, upload=440470]: Done begin upload segment. Dec 26 13:22:32.680452900 (501h53m32.551535935s) [noise=true , address=77.90.31.13:4627 , piece_id=SHXBCOH4RVCH3LTMDY6T2FOXQUY42MFFQL6QUR7N6LZL65PF56NA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.680490111 (501h53m32.551573176s) [noise=true , address=77.90.31.16:4735 , piece_id=ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.681369505 (501h53m32.55245254s) [noise=true , address=145.79.227.28:4727 , piece_id=SA3AYCPVMMTMD5GQKBNSRSVSDBN6APHPLRSVUTLIKA3YAJFCIJHA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.682075288 (501h53m32.553158323s) [noise=true , address=145.79.227.13:4802 , piece_id=PVACSPBULCNEYH3DED2VIIPVB4FPRBBEYVNBEH4VQWHH3BPNWGBA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.683095442 (501h53m32.554178477s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.684650502 (501h53m32.555733537s) [noise=true , address=145.79.227.19:4078 , piece_id=XIGHVBQOBXXURFLGSQC54WCV3PT6IS2VKNBX3IIMDNFF7UM2UT7Q, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.685456448 (501h53m32.556539483s) [noise=true , address=145.79.227.20:4438 , piece_id=ASLXFWRJUW4BHSX2PKZK4JIJJDRDXIRCAG2VXQCTUR62NP76GO7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.686754629 (501h53m32.557837664s) [noise=true , address=45.91.116.8:4045 , piece_id=TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.687518043 (501h53m32.558601078s) [noise=true , address=145.79.227.38:4978 , piece_id=JLOJJ5K6MLTKDKT2EBDZGWAY6HJQCID4UBNHZAOVNXELP5YVHU7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.693524029 (501h53m32.564607064s) [noise=true , address=145.79.227.45:4231 , piece_id=XZIGODMIDD5BQFXINLEUHZEON6BQANHSLIETFV55UC4NVMD7E5GA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694151704 (501h53m32.565234739s) [noise=true , address=145.79.227.37:4567 , piece_id=LTIMLA6I4ZJLYOTUGGROMJ7N3IAQ7XYLDFCUMJWPJ4UHPK32GLQQ, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694519495 (501h53m32.56560253s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694951247 (501h53m32.566034282s) [noise=true , address=145.79.227.39:4015 , piece_id=DI2J3QCAO55CGHNSEHVMXU46I4P77NZNGX5MW3I2DN6ISYJ4JT3A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.695331851 (501h53m32.566414896s) [noise=true , address=145.79.227.6:4258 , piece_id=HKLVNVOO3MIZ6KUNAHQGMRQSGFA37TKA54N34HAYRFRUGWPKCQ6Q, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.695741902 (501h53m32.566824948s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.696147946 (501h53m32.567230991s) [noise=true , address=45.91.116.6:4976 , piece_id=PYPVPYVEI6D22PVBBB37EMWUEXM3YKJ5BTIL7MAMXUAX5AL2XVPA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:34.406954778 (501h53m34.278037813s) [upload=440470]: Next returned nil segment Dec 26 13:22:34.406968063 (501h53m34.278051098s) [upload=440470]: Waiting for error group managing segments... Dec 26 13:22:34.696844389 (501h53m34.567927424s) [noise=true , address=145.79.227.28:4727 , piece_id=SA3AYCPVMMTMD5GQKBNSRSVSDBN6APHPLRSVUTLIKA3YAJFCIJHA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.704462957 (501h53m34.575546002s) [noise=true , address=145.79.227.38:4978 , piece_id=JLOJJ5K6MLTKDKT2EBDZGWAY6HJQCID4UBNHZAOVNXELP5YVHU7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.706884898 (501h53m34.577967943s) [noise=true , address=145.79.227.13:4802 , piece_id=PVACSPBULCNEYH3DED2VIIPVB4FPRBBEYVNBEH4VQWHH3BPNWGBA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.710527174 (501h53m34.581610219s) [noise=true , address=145.79.227.6:4258 , piece_id=HKLVNVOO3MIZ6KUNAHQGMRQSGFA37TKA54N34HAYRFRUGWPKCQ6Q, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.716355412 (501h53m34.587438487s) [noise=true , address=145.79.227.45:4231 , piece_id=XZIGODMIDD5BQFXINLEUHZEON6BQANHSLIETFV55UC4NVMD7E5GA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.722119418 (501h53m34.593202463s) [noise=true , address=145.79.227.20:4438 , piece_id=ASLXFWRJUW4BHSX2PKZK4JIJJDRDXIRCAG2VXQCTUR62NP76GO7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.722645529 (501h53m34.593728575s) [noise=true , address=145.79.227.19:4078 , piece_id=XIGHVBQOBXXURFLGSQC54WCV3PT6IS2VKNBX3IIMDNFF7UM2UT7Q, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.731926392 (501h53m34.603009437s) [noise=true , address=145.79.227.37:4567 , piece_id=LTIMLA6I4ZJLYOTUGGROMJ7N3IAQ7XYLDFCUMJWPJ4UHPK32GLQQ, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.741636291 (501h53m34.612719336s) [noise=true , address=145.79.227.39:4015 , piece_id=DI2J3QCAO55CGHNSEHVMXU46I4P77NZNGX5MW3I2DN6ISYJ4JT3A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.966402947 (501h53m34.837485992s) [noise=true , address=45.91.116.6:4976 , piece_id=PYPVPYVEI6D22PVBBB37EMWUEXM3YKJ5BTIL7MAMXUAX5AL2XVPA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.200199657 (501h53m35.071282702s) [noise=true , address=77.90.31.13:4627 , piece_id=SHXBCOH4RVCH3LTMDY6T2FOXQUY42MFFQL6QUR7N6LZL65PF56NA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.482129723 (501h53m35.353212758s) [noise=true , address=45.91.116.8:4045 , piece_id=TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.673121025 (501h53m35.54420406s) [noise=true , address=77.90.31.16:4735 , piece_id=ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.673140882 (501h53m35.544223917s) [seg_pos={0 0}, upload=440470]: Segment reached optimal threshold of 13 pieces. Dec 26 13:22:35.673324491 (501h53m35.544407536s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673332608 (501h53m35.544415653s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673511408 (501h53m35.544594443s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673517529 (501h53m35.544600574s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673559619 (501h53m35.544642654s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673580289 (501h53m35.544663334s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673627980 (501h53m35.544711025s) [seg_pos={0 0}, upload=440470]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:22:35.673776222 (501h53m35.544859277s) [upload=440470]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:22:35.673784368 (501h53m35.544867413s) [upload=440470]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:22:35.965152704 (501h53m35.836235749s) [upload=440470]: Done uploading segments... Dec 26 13:22:35.965174246 (501h53m35.836257291s) [upload=440470]: Upload finished. err: <nil> Duration: 6.162340318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........X.XXX...X..X...................XX.X........X.XXX.] duration: 34.915138482s trace: https://jaeger.dev.storj.io/trace/7e20d121f5044dca FAIL (2025-12-26 13:24:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa, Piece ID: OOOMD4SJ5XYCBCVDXUJMUTHOTQNIPUOXL6WHEJPMZW5DEJI2FJCQ) context deadline exceeded; piecestore: (Node ID: 12YEcohZBsQSm45orPuT8np4EGgXQaWAM8rCsmUfA1PfNubwLmo, Piece ID: YVMMXYCWCY2LNM2NAT3YL6S23FA3MON66T7RSZVVOSMXTW3I26GQ) context deadline exceeded; piecestore: (Node ID: 12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, Piece ID: Z63CS756UG7KW54Q7HYQSL5ZS3ILZAZ5DDX777YKYV4C3FQPROJA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: LWAKLVFTKBF6H27TDJ3WHI7I44QY4EHN3KA24V5DAIUXEUIOLWMQ) context deadline exceeded; piecestore: (Node ID: 1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf, Piece ID: KTYRS7YEU6NDXC6HHQY6VZCQ2OPZM5F6TBGEW6JKA42ONEPSFIAQ) context deadline exceeded ======LOGS====== 4 (501h53m58.242694919s) [noise=true , address=145.79.227.30:4776 , piece_id=CO7WME62ULWJVH2YJVA5QZFQCR443XDKVQ7QHOMLN2LBAECLADZQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.371955679 (501h53m58.243038714s) [noise=true , address=77.90.31.19:4847 , piece_id=WOZGWAU4P4N777LO76YC3B75I5RLSJYEC2S2X5ZWV5VG5KI2JP2A, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.372263385 (501h53m58.24334642s) [noise=true , address=77.90.31.18:4840 , piece_id=LWAKLVFTKBF6H27TDJ3WHI7I44QY4EHN3KA24V5DAIUXEUIOLWMQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.372590277 (501h53m58.243673312s) [noise=true , address=64.49.13.27:4959 , piece_id=RABCT4RXOJZKNHKL4P3LQZVN3NRH35R7QQOVQPCTVRKGAYCBVDKQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.372942869 (501h53m58.244025904s) [noise=true , address=45.91.116.14:4210 , piece_id=WX4CQRQ7OPPJ4HX56FXMGSKFB6YPAUOZF7AZPAW3OQBLSO5YTJOQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.373352960 (501h53m58.244435995s) [noise=true , address=77.90.31.12:4594 , piece_id=TASFEX7RZ54F3EAJOCZYFZ4MW5SGMCJBKP5CGQ7VUUYHFTGX22UQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.373871137 (501h53m58.244954182s) [noise=true , address=145.79.227.40:4076 , piece_id=TVGJ6U2B66Q3KCHLC432ALEG7ZTW4FEP5RFM6GS3TVOF5Z2F6O6A, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.374328908 (501h53m58.245411943s) [noise=true , address=77.90.31.11:4564 , piece_id=KCI3COEBMLWCPMIXD4OXQDP7BVTPKBPNK33KZHEKLVU3RCMSGJUQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.374708160 (501h53m58.245791195s) [noise=true , address=77.90.31.15:4724 , piece_id=YVMMXYCWCY2LNM2NAT3YL6S23FA3MON66T7RSZVVOSMXTW3I26GQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.375193164 (501h53m58.246276199s) [noise=true , address=64.49.13.169:4710 , piece_id=2G2ZMEVMHQCS6Q6EECWW253P6W2DU4GCEKASIOSFL6HYCNSV223A, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.375685752 (501h53m58.246768797s) [noise=true , address=145.79.227.34:4932 , piece_id=VOJ5JVKDGFEUQJ4KJAVSHDBDLAWO6W5FS2ZY43EJH2Y6WT5W4EUQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.376142111 (501h53m58.247225146s) [noise=true , address=45.91.116.13:4165 , piece_id=Z63CS756UG7KW54Q7HYQSL5ZS3ILZAZ5DDX777YKYV4C3FQPROJA, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.377297270 (501h53m58.248380305s) [noise=true , address=77.90.31.14:4666 , piece_id=OGEQ4WMR234W5OMIR6EGLSAGDSDOBNLCA5GNNLEPQK5OGQOD72XA, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.378952673 (501h53m58.250035718s) [noise=true , address=45.91.116.10:4115 , piece_id=KTYRS7YEU6NDXC6HHQY6VZCQ2OPZM5F6TBGEW6JKA42ONEPSFIAQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.384863739 (501h53m58.255946774s) [noise=true , address=145.79.227.44:4214 , piece_id=FHCFNB3O4QYDADZW5Q2AIZHDL5OJHJHJPKROLLVZVADJSOM4FZOA, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.385445988 (501h53m58.256529023s) [noise=true , address=64.49.13.140:4508 , piece_id=DZGCEKUD5MHWSI2UKZUU6GIH7VT47MMQLJGWTJI6WGAJPNG7GOHQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:59.702585566 (501h53m59.573668612s) [upload=440475]: Next returned nil segment Dec 26 13:22:59.702597689 (501h53m59.573680724s) [upload=440475]: Waiting for error group managing segments... Dec 26 13:23:00.046836491 (501h53m59.917919527s) [noise=true , address=64.49.13.140:4508 , piece_id=DZGCEKUD5MHWSI2UKZUU6GIH7VT47MMQLJGWTJI6WGAJPNG7GOHQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.105508363 (501h53m59.976591398s) [noise=true , address=64.49.13.27:4959 , piece_id=RABCT4RXOJZKNHKL4P3LQZVN3NRH35R7QQOVQPCTVRKGAYCBVDKQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.277388944 (501h54m0.148471989s) [noise=true , address=145.79.227.40:4076 , piece_id=TVGJ6U2B66Q3KCHLC432ALEG7ZTW4FEP5RFM6GS3TVOF5Z2F6O6A, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.464314876 (501h54m0.335397921s) [noise=true , address=77.90.31.21:4892 , piece_id=OOOMD4SJ5XYCBCVDXUJMUTHOTQNIPUOXL6WHEJPMZW5DEJI2FJCQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.465241469 (501h54m0.336324514s) [noise=true , address=77.90.31.19:4847 , piece_id=WOZGWAU4P4N777LO76YC3B75I5RLSJYEC2S2X5ZWV5VG5KI2JP2A, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.481001089 (501h54m0.352084134s) [noise=true , address=77.90.31.12:4594 , piece_id=TASFEX7RZ54F3EAJOCZYFZ4MW5SGMCJBKP5CGQ7VUUYHFTGX22UQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.489735561 (501h54m0.360818606s) [noise=true , address=77.90.31.11:4564 , piece_id=KCI3COEBMLWCPMIXD4OXQDP7BVTPKBPNK33KZHEKLVU3RCMSGJUQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.535886941 (501h54m0.406969976s) [noise=true , address=77.90.31.14:4666 , piece_id=OGEQ4WMR234W5OMIR6EGLSAGDSDOBNLCA5GNNLEPQK5OGQOD72XA, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.556984374 (501h54m0.428067409s) [noise=true , address=45.91.116.10:4115 , piece_id=KTYRS7YEU6NDXC6HHQY6VZCQ2OPZM5F6TBGEW6JKA42ONEPSFIAQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.756584780 (501h54m0.627667825s) [noise=true , address=77.90.31.18:4840 , piece_id=LWAKLVFTKBF6H27TDJ3WHI7I44QY4EHN3KA24V5DAIUXEUIOLWMQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.759223324 (501h54m0.630306369s) [noise=true , address=77.90.31.15:4724 , piece_id=YVMMXYCWCY2LNM2NAT3YL6S23FA3MON66T7RSZVVOSMXTW3I26GQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.793528497 (501h54m0.664611532s) [noise=true , address=45.91.116.13:4165 , piece_id=Z63CS756UG7KW54Q7HYQSL5ZS3ILZAZ5DDX777YKYV4C3FQPROJA, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:01.163817954 (501h54m1.03490099s) [noise=true , address=145.79.227.34:4932 , piece_id=VOJ5JVKDGFEUQJ4KJAVSHDBDLAWO6W5FS2ZY43EJH2Y6WT5W4EUQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:01.237083525 (501h54m1.10816657s) [noise=true , address=145.79.227.30:4776 , piece_id=CO7WME62ULWJVH2YJVA5QZFQCR443XDKVQ7QHOMLN2LBAECLADZQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:01.237114474 (501h54m1.108197519s) [seg_pos={0 0}, upload=440475]: Segment reached optimal threshold of 14 pieces. Dec 26 13:23:01.237405388 (501h54m1.108488433s) [noise=true , address=145.79.227.44:4214 , piece_id=FHCFNB3O4QYDADZW5Q2AIZHDL5OJHJHJPKROLLVZVADJSOM4FZOA, seg_pos={0 0}, upload=440475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 26 13:23:01.237415407 (501h54m1.108498452s) [noise=true , address=145.79.227.44:4214 , piece_id=FHCFNB3O4QYDADZW5Q2AIZHDL5OJHJHJPKROLLVZVADJSOM4FZOA, seg_pos={0 0}, upload=440475]: Long tail cancellation due to optimal threshold Dec 26 13:23:01.237481613 (501h54m1.108564658s) [noise=true , address=45.91.116.14:4210 , piece_id=WX4CQRQ7OPPJ4HX56FXMGSKFB6YPAUOZF7AZPAW3OQBLSO5YTJOQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 13:23:01.237507573 (501h54m1.108590618s) [noise=true , address=45.91.116.14:4210 , piece_id=WX4CQRQ7OPPJ4HX56FXMGSKFB6YPAUOZF7AZPAW3OQBLSO5YTJOQ, seg_pos={0 0}, upload=440475]: Long tail cancellation due to optimal threshold Dec 26 13:23:01.237565684 (501h54m1.108648729s) [noise=true , address=64.49.13.169:4710 , piece_id=2G2ZMEVMHQCS6Q6EECWW253P6W2DU4GCEKASIOSFL6HYCNSV223A, seg_pos={0 0}, upload=440475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:40550->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:40550->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40550->64.49.13.169:4710: use of closed network connection Dec 26 13:23:01.237573538 (501h54m1.108656593s) [noise=true , address=64.49.13.169:4710 , piece_id=2G2ZMEVMHQCS6Q6EECWW253P6W2DU4GCEKASIOSFL6HYCNSV223A, seg_pos={0 0}, upload=440475]: Long tail cancellation due to optimal threshold Dec 26 13:23:01.237827592 (501h54m1.108910637s) [noise=true , address=77.90.31.10:4549 , piece_id=47XAZGCBBB3UZCN3KMFFKICD6IEJWYU2GUS2BZIIUL4O6SNOCFAA, seg_pos={0 0}, upload=440475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 13:23:01.237835807 (501h54m1.108918842s) [noise=true , address=77.90.31.10:4549 , piece_id=47XAZGCBBB3UZCN3KMFFKICD6IEJWYU2GUS2BZIIUL4O6SNOCFAA, seg_pos={0 0}, upload=440475]: Long tail cancellation due to optimal threshold Dec 26 13:23:01.237881936 (501h54m1.108964971s) [seg_pos={0 0}, upload=440475]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:23:01.238179141 (501h54m1.109262186s) [upload=440475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:23:01.238185533 (501h54m1.109268568s) [upload=440475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:23:01.536618132 (501h54m1.407701177s) [upload=440475]: Done uploading segments... Dec 26 13:23:01.536643641 (501h54m1.407726686s) [upload=440475]: Upload finished. err: <nil> Duration: 34.915155023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.X] duration: 5.00146881s trace: https://jaeger.dev.storj.io/trace/54f7c37e2de460db FAIL (2025-12-26 13:24:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001492415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [............X.XXX...X..X...................XX.X........X.XXX] duration: 1m0.003505105s trace: https://jaeger.dev.storj.io/trace/7e20d121f5044dca FAIL (2025-12-26 13:23:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa, Piece ID: OOOMD4SJ5XYCBCVDXUJMUTHOTQNIPUOXL6WHEJPMZW5DEJI2FJCQ) context deadline exceeded; piecestore: (Node ID: 12YEcohZBsQSm45orPuT8np4EGgXQaWAM8rCsmUfA1PfNubwLmo, Piece ID: YVMMXYCWCY2LNM2NAT3YL6S23FA3MON66T7RSZVVOSMXTW3I26GQ) context deadline exceeded; piecestore: (Node ID: 12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, Piece ID: Z63CS756UG7KW54Q7HYQSL5ZS3ILZAZ5DDX777YKYV4C3FQPROJA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: LWAKLVFTKBF6H27TDJ3WHI7I44QY4EHN3KA24V5DAIUXEUIOLWMQ) context deadline exceeded; piecestore: (Node ID: 1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf, Piece ID: KTYRS7YEU6NDXC6HHQY6VZCQ2OPZM5F6TBGEW6JKA42ONEPSFIAQ) context deadline exceeded ======LOGS====== 4 (501h53m58.242694919s) [noise=true , address=145.79.227.30:4776 , piece_id=CO7WME62ULWJVH2YJVA5QZFQCR443XDKVQ7QHOMLN2LBAECLADZQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.371955679 (501h53m58.243038714s) [noise=true , address=77.90.31.19:4847 , piece_id=WOZGWAU4P4N777LO76YC3B75I5RLSJYEC2S2X5ZWV5VG5KI2JP2A, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.372263385 (501h53m58.24334642s) [noise=true , address=77.90.31.18:4840 , piece_id=LWAKLVFTKBF6H27TDJ3WHI7I44QY4EHN3KA24V5DAIUXEUIOLWMQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.372590277 (501h53m58.243673312s) [noise=true , address=64.49.13.27:4959 , piece_id=RABCT4RXOJZKNHKL4P3LQZVN3NRH35R7QQOVQPCTVRKGAYCBVDKQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.372942869 (501h53m58.244025904s) [noise=true , address=45.91.116.14:4210 , piece_id=WX4CQRQ7OPPJ4HX56FXMGSKFB6YPAUOZF7AZPAW3OQBLSO5YTJOQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.373352960 (501h53m58.244435995s) [noise=true , address=77.90.31.12:4594 , piece_id=TASFEX7RZ54F3EAJOCZYFZ4MW5SGMCJBKP5CGQ7VUUYHFTGX22UQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.373871137 (501h53m58.244954182s) [noise=true , address=145.79.227.40:4076 , piece_id=TVGJ6U2B66Q3KCHLC432ALEG7ZTW4FEP5RFM6GS3TVOF5Z2F6O6A, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.374328908 (501h53m58.245411943s) [noise=true , address=77.90.31.11:4564 , piece_id=KCI3COEBMLWCPMIXD4OXQDP7BVTPKBPNK33KZHEKLVU3RCMSGJUQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.374708160 (501h53m58.245791195s) [noise=true , address=77.90.31.15:4724 , piece_id=YVMMXYCWCY2LNM2NAT3YL6S23FA3MON66T7RSZVVOSMXTW3I26GQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.375193164 (501h53m58.246276199s) [noise=true , address=64.49.13.169:4710 , piece_id=2G2ZMEVMHQCS6Q6EECWW253P6W2DU4GCEKASIOSFL6HYCNSV223A, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.375685752 (501h53m58.246768797s) [noise=true , address=145.79.227.34:4932 , piece_id=VOJ5JVKDGFEUQJ4KJAVSHDBDLAWO6W5FS2ZY43EJH2Y6WT5W4EUQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.376142111 (501h53m58.247225146s) [noise=true , address=45.91.116.13:4165 , piece_id=Z63CS756UG7KW54Q7HYQSL5ZS3ILZAZ5DDX777YKYV4C3FQPROJA, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.377297270 (501h53m58.248380305s) [noise=true , address=77.90.31.14:4666 , piece_id=OGEQ4WMR234W5OMIR6EGLSAGDSDOBNLCA5GNNLEPQK5OGQOD72XA, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.378952673 (501h53m58.250035718s) [noise=true , address=45.91.116.10:4115 , piece_id=KTYRS7YEU6NDXC6HHQY6VZCQ2OPZM5F6TBGEW6JKA42ONEPSFIAQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.384863739 (501h53m58.255946774s) [noise=true , address=145.79.227.44:4214 , piece_id=FHCFNB3O4QYDADZW5Q2AIZHDL5OJHJHJPKROLLVZVADJSOM4FZOA, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:58.385445988 (501h53m58.256529023s) [noise=true , address=64.49.13.140:4508 , piece_id=DZGCEKUD5MHWSI2UKZUU6GIH7VT47MMQLJGWTJI6WGAJPNG7GOHQ, seg_pos={0 0}, upload=440475]: Uploading piece... Dec 26 13:22:59.702585566 (501h53m59.573668612s) [upload=440475]: Next returned nil segment Dec 26 13:22:59.702597689 (501h53m59.573680724s) [upload=440475]: Waiting for error group managing segments... Dec 26 13:23:00.046836491 (501h53m59.917919527s) [noise=true , address=64.49.13.140:4508 , piece_id=DZGCEKUD5MHWSI2UKZUU6GIH7VT47MMQLJGWTJI6WGAJPNG7GOHQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.105508363 (501h53m59.976591398s) [noise=true , address=64.49.13.27:4959 , piece_id=RABCT4RXOJZKNHKL4P3LQZVN3NRH35R7QQOVQPCTVRKGAYCBVDKQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.277388944 (501h54m0.148471989s) [noise=true , address=145.79.227.40:4076 , piece_id=TVGJ6U2B66Q3KCHLC432ALEG7ZTW4FEP5RFM6GS3TVOF5Z2F6O6A, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.464314876 (501h54m0.335397921s) [noise=true , address=77.90.31.21:4892 , piece_id=OOOMD4SJ5XYCBCVDXUJMUTHOTQNIPUOXL6WHEJPMZW5DEJI2FJCQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.465241469 (501h54m0.336324514s) [noise=true , address=77.90.31.19:4847 , piece_id=WOZGWAU4P4N777LO76YC3B75I5RLSJYEC2S2X5ZWV5VG5KI2JP2A, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.481001089 (501h54m0.352084134s) [noise=true , address=77.90.31.12:4594 , piece_id=TASFEX7RZ54F3EAJOCZYFZ4MW5SGMCJBKP5CGQ7VUUYHFTGX22UQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.489735561 (501h54m0.360818606s) [noise=true , address=77.90.31.11:4564 , piece_id=KCI3COEBMLWCPMIXD4OXQDP7BVTPKBPNK33KZHEKLVU3RCMSGJUQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.535886941 (501h54m0.406969976s) [noise=true , address=77.90.31.14:4666 , piece_id=OGEQ4WMR234W5OMIR6EGLSAGDSDOBNLCA5GNNLEPQK5OGQOD72XA, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.556984374 (501h54m0.428067409s) [noise=true , address=45.91.116.10:4115 , piece_id=KTYRS7YEU6NDXC6HHQY6VZCQ2OPZM5F6TBGEW6JKA42ONEPSFIAQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.756584780 (501h54m0.627667825s) [noise=true , address=77.90.31.18:4840 , piece_id=LWAKLVFTKBF6H27TDJ3WHI7I44QY4EHN3KA24V5DAIUXEUIOLWMQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.759223324 (501h54m0.630306369s) [noise=true , address=77.90.31.15:4724 , piece_id=YVMMXYCWCY2LNM2NAT3YL6S23FA3MON66T7RSZVVOSMXTW3I26GQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:00.793528497 (501h54m0.664611532s) [noise=true , address=45.91.116.13:4165 , piece_id=Z63CS756UG7KW54Q7HYQSL5ZS3ILZAZ5DDX777YKYV4C3FQPROJA, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:01.163817954 (501h54m1.03490099s) [noise=true , address=145.79.227.34:4932 , piece_id=VOJ5JVKDGFEUQJ4KJAVSHDBDLAWO6W5FS2ZY43EJH2Y6WT5W4EUQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:01.237083525 (501h54m1.10816657s) [noise=true , address=145.79.227.30:4776 , piece_id=CO7WME62ULWJVH2YJVA5QZFQCR443XDKVQ7QHOMLN2LBAECLADZQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: <nil> Dec 26 13:23:01.237114474 (501h54m1.108197519s) [seg_pos={0 0}, upload=440475]: Segment reached optimal threshold of 14 pieces. Dec 26 13:23:01.237405388 (501h54m1.108488433s) [noise=true , address=145.79.227.44:4214 , piece_id=FHCFNB3O4QYDADZW5Q2AIZHDL5OJHJHJPKROLLVZVADJSOM4FZOA, seg_pos={0 0}, upload=440475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 26 13:23:01.237415407 (501h54m1.108498452s) [noise=true , address=145.79.227.44:4214 , piece_id=FHCFNB3O4QYDADZW5Q2AIZHDL5OJHJHJPKROLLVZVADJSOM4FZOA, seg_pos={0 0}, upload=440475]: Long tail cancellation due to optimal threshold Dec 26 13:23:01.237481613 (501h54m1.108564658s) [noise=true , address=45.91.116.14:4210 , piece_id=WX4CQRQ7OPPJ4HX56FXMGSKFB6YPAUOZF7AZPAW3OQBLSO5YTJOQ, seg_pos={0 0}, upload=440475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 13:23:01.237507573 (501h54m1.108590618s) [noise=true , address=45.91.116.14:4210 , piece_id=WX4CQRQ7OPPJ4HX56FXMGSKFB6YPAUOZF7AZPAW3OQBLSO5YTJOQ, seg_pos={0 0}, upload=440475]: Long tail cancellation due to optimal threshold Dec 26 13:23:01.237565684 (501h54m1.108648729s) [noise=true , address=64.49.13.169:4710 , piece_id=2G2ZMEVMHQCS6Q6EECWW253P6W2DU4GCEKASIOSFL6HYCNSV223A, seg_pos={0 0}, upload=440475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:40550->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:40550->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40550->64.49.13.169:4710: use of closed network connection Dec 26 13:23:01.237573538 (501h54m1.108656593s) [noise=true , address=64.49.13.169:4710 , piece_id=2G2ZMEVMHQCS6Q6EECWW253P6W2DU4GCEKASIOSFL6HYCNSV223A, seg_pos={0 0}, upload=440475]: Long tail cancellation due to optimal threshold Dec 26 13:23:01.237827592 (501h54m1.108910637s) [noise=true , address=77.90.31.10:4549 , piece_id=47XAZGCBBB3UZCN3KMFFKICD6IEJWYU2GUS2BZIIUL4O6SNOCFAA, seg_pos={0 0}, upload=440475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 13:23:01.237835807 (501h54m1.108918842s) [noise=true , address=77.90.31.10:4549 , piece_id=47XAZGCBBB3UZCN3KMFFKICD6IEJWYU2GUS2BZIIUL4O6SNOCFAA, seg_pos={0 0}, upload=440475]: Long tail cancellation due to optimal threshold Dec 26 13:23:01.237881936 (501h54m1.108964971s) [seg_pos={0 0}, upload=440475]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:23:01.238179141 (501h54m1.109262186s) [upload=440475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:23:01.238185533 (501h54m1.109268568s) [upload=440475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:23:01.536618132 (501h54m1.407701177s) [upload=440475]: Done uploading segments... Dec 26 13:23:01.536643641 (501h54m1.407726686s) [upload=440475]: Upload finished. err: <nil> Duration: 1m0.003516468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [..XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.] duration: 4.974271271s trace: https://jaeger.dev.storj.io/trace/cde8d6d2d0dcc52 FAIL (2025-12-26 13:23:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.974406939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X......X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X.] duration: 5.741666879s trace: https://jaeger.dev.storj.io/trace/70ecae77a4804cc FAIL (2025-12-26 13:23:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA) context deadline exceeded ======LOGS====== Dec 26 13:22:32.436256516 (501h53m32.307339561s) [upload=440470]: Starting upload Dec 26 13:22:32.436335557 (501h53m32.307418622s) [upload=440470]: Done starting upload Dec 26 13:22:32.436536680 (501h53m32.307619725s) [upload=440470]: Uploading segments... Dec 26 13:22:32.436601352 (501h53m32.307684407s) [upload=440470]: Got next segment. Inline: false Dec 26 13:22:32.436635207 (501h53m32.307718242s) [upload=440470]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:22:32.436641259 (501h53m32.307724304s) [upload=440470]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:22:32.680370322 (501h53m32.551453357s) [seg_pos={0 0}, upload=440470]: Begin upload segment... Dec 26 13:22:32.680425708 (501h53m32.551508743s) [seg_pos={0 0}, upload=440470]: Done begin upload segment. Dec 26 13:22:32.680452900 (501h53m32.551535935s) [noise=true , address=77.90.31.13:4627 , piece_id=SHXBCOH4RVCH3LTMDY6T2FOXQUY42MFFQL6QUR7N6LZL65PF56NA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.680490111 (501h53m32.551573176s) [noise=true , address=77.90.31.16:4735 , piece_id=ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.681369505 (501h53m32.55245254s) [noise=true , address=145.79.227.28:4727 , piece_id=SA3AYCPVMMTMD5GQKBNSRSVSDBN6APHPLRSVUTLIKA3YAJFCIJHA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.682075288 (501h53m32.553158323s) [noise=true , address=145.79.227.13:4802 , piece_id=PVACSPBULCNEYH3DED2VIIPVB4FPRBBEYVNBEH4VQWHH3BPNWGBA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.683095442 (501h53m32.554178477s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.684650502 (501h53m32.555733537s) [noise=true , address=145.79.227.19:4078 , piece_id=XIGHVBQOBXXURFLGSQC54WCV3PT6IS2VKNBX3IIMDNFF7UM2UT7Q, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.685456448 (501h53m32.556539483s) [noise=true , address=145.79.227.20:4438 , piece_id=ASLXFWRJUW4BHSX2PKZK4JIJJDRDXIRCAG2VXQCTUR62NP76GO7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.686754629 (501h53m32.557837664s) [noise=true , address=45.91.116.8:4045 , piece_id=TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.687518043 (501h53m32.558601078s) [noise=true , address=145.79.227.38:4978 , piece_id=JLOJJ5K6MLTKDKT2EBDZGWAY6HJQCID4UBNHZAOVNXELP5YVHU7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.693524029 (501h53m32.564607064s) [noise=true , address=145.79.227.45:4231 , piece_id=XZIGODMIDD5BQFXINLEUHZEON6BQANHSLIETFV55UC4NVMD7E5GA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694151704 (501h53m32.565234739s) [noise=true , address=145.79.227.37:4567 , piece_id=LTIMLA6I4ZJLYOTUGGROMJ7N3IAQ7XYLDFCUMJWPJ4UHPK32GLQQ, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694519495 (501h53m32.56560253s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694951247 (501h53m32.566034282s) [noise=true , address=145.79.227.39:4015 , piece_id=DI2J3QCAO55CGHNSEHVMXU46I4P77NZNGX5MW3I2DN6ISYJ4JT3A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.695331851 (501h53m32.566414896s) [noise=true , address=145.79.227.6:4258 , piece_id=HKLVNVOO3MIZ6KUNAHQGMRQSGFA37TKA54N34HAYRFRUGWPKCQ6Q, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.695741902 (501h53m32.566824948s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.696147946 (501h53m32.567230991s) [noise=true , address=45.91.116.6:4976 , piece_id=PYPVPYVEI6D22PVBBB37EMWUEXM3YKJ5BTIL7MAMXUAX5AL2XVPA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:34.406954778 (501h53m34.278037813s) [upload=440470]: Next returned nil segment Dec 26 13:22:34.406968063 (501h53m34.278051098s) [upload=440470]: Waiting for error group managing segments... Dec 26 13:22:34.696844389 (501h53m34.567927424s) [noise=true , address=145.79.227.28:4727 , piece_id=SA3AYCPVMMTMD5GQKBNSRSVSDBN6APHPLRSVUTLIKA3YAJFCIJHA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.704462957 (501h53m34.575546002s) [noise=true , address=145.79.227.38:4978 , piece_id=JLOJJ5K6MLTKDKT2EBDZGWAY6HJQCID4UBNHZAOVNXELP5YVHU7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.706884898 (501h53m34.577967943s) [noise=true , address=145.79.227.13:4802 , piece_id=PVACSPBULCNEYH3DED2VIIPVB4FPRBBEYVNBEH4VQWHH3BPNWGBA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.710527174 (501h53m34.581610219s) [noise=true , address=145.79.227.6:4258 , piece_id=HKLVNVOO3MIZ6KUNAHQGMRQSGFA37TKA54N34HAYRFRUGWPKCQ6Q, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.716355412 (501h53m34.587438487s) [noise=true , address=145.79.227.45:4231 , piece_id=XZIGODMIDD5BQFXINLEUHZEON6BQANHSLIETFV55UC4NVMD7E5GA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.722119418 (501h53m34.593202463s) [noise=true , address=145.79.227.20:4438 , piece_id=ASLXFWRJUW4BHSX2PKZK4JIJJDRDXIRCAG2VXQCTUR62NP76GO7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.722645529 (501h53m34.593728575s) [noise=true , address=145.79.227.19:4078 , piece_id=XIGHVBQOBXXURFLGSQC54WCV3PT6IS2VKNBX3IIMDNFF7UM2UT7Q, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.731926392 (501h53m34.603009437s) [noise=true , address=145.79.227.37:4567 , piece_id=LTIMLA6I4ZJLYOTUGGROMJ7N3IAQ7XYLDFCUMJWPJ4UHPK32GLQQ, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.741636291 (501h53m34.612719336s) [noise=true , address=145.79.227.39:4015 , piece_id=DI2J3QCAO55CGHNSEHVMXU46I4P77NZNGX5MW3I2DN6ISYJ4JT3A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.966402947 (501h53m34.837485992s) [noise=true , address=45.91.116.6:4976 , piece_id=PYPVPYVEI6D22PVBBB37EMWUEXM3YKJ5BTIL7MAMXUAX5AL2XVPA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.200199657 (501h53m35.071282702s) [noise=true , address=77.90.31.13:4627 , piece_id=SHXBCOH4RVCH3LTMDY6T2FOXQUY42MFFQL6QUR7N6LZL65PF56NA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.482129723 (501h53m35.353212758s) [noise=true , address=45.91.116.8:4045 , piece_id=TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.673121025 (501h53m35.54420406s) [noise=true , address=77.90.31.16:4735 , piece_id=ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.673140882 (501h53m35.544223917s) [seg_pos={0 0}, upload=440470]: Segment reached optimal threshold of 13 pieces. Dec 26 13:22:35.673324491 (501h53m35.544407536s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673332608 (501h53m35.544415653s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673511408 (501h53m35.544594443s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673517529 (501h53m35.544600574s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673559619 (501h53m35.544642654s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673580289 (501h53m35.544663334s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673627980 (501h53m35.544711025s) [seg_pos={0 0}, upload=440470]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:22:35.673776222 (501h53m35.544859277s) [upload=440470]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:22:35.673784368 (501h53m35.544867413s) [upload=440470]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:22:35.965152704 (501h53m35.836235749s) [upload=440470]: Done uploading segments... Dec 26 13:22:35.965174246 (501h53m35.836257291s) [upload=440470]: Upload finished. err: <nil> Duration: 5.741707416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X......X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X] duration: 30.004455152s trace: https://jaeger.dev.storj.io/trace/70ecae77a4804cc FAIL (2025-12-26 13:23:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA) context deadline exceeded ======LOGS====== Dec 26 13:22:32.436256516 (501h53m32.307339561s) [upload=440470]: Starting upload Dec 26 13:22:32.436335557 (501h53m32.307418622s) [upload=440470]: Done starting upload Dec 26 13:22:32.436536680 (501h53m32.307619725s) [upload=440470]: Uploading segments... Dec 26 13:22:32.436601352 (501h53m32.307684407s) [upload=440470]: Got next segment. Inline: false Dec 26 13:22:32.436635207 (501h53m32.307718242s) [upload=440470]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:22:32.436641259 (501h53m32.307724304s) [upload=440470]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:22:32.680370322 (501h53m32.551453357s) [seg_pos={0 0}, upload=440470]: Begin upload segment... Dec 26 13:22:32.680425708 (501h53m32.551508743s) [seg_pos={0 0}, upload=440470]: Done begin upload segment. Dec 26 13:22:32.680452900 (501h53m32.551535935s) [noise=true , address=77.90.31.13:4627 , piece_id=SHXBCOH4RVCH3LTMDY6T2FOXQUY42MFFQL6QUR7N6LZL65PF56NA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.680490111 (501h53m32.551573176s) [noise=true , address=77.90.31.16:4735 , piece_id=ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.681369505 (501h53m32.55245254s) [noise=true , address=145.79.227.28:4727 , piece_id=SA3AYCPVMMTMD5GQKBNSRSVSDBN6APHPLRSVUTLIKA3YAJFCIJHA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.682075288 (501h53m32.553158323s) [noise=true , address=145.79.227.13:4802 , piece_id=PVACSPBULCNEYH3DED2VIIPVB4FPRBBEYVNBEH4VQWHH3BPNWGBA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.683095442 (501h53m32.554178477s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.684650502 (501h53m32.555733537s) [noise=true , address=145.79.227.19:4078 , piece_id=XIGHVBQOBXXURFLGSQC54WCV3PT6IS2VKNBX3IIMDNFF7UM2UT7Q, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.685456448 (501h53m32.556539483s) [noise=true , address=145.79.227.20:4438 , piece_id=ASLXFWRJUW4BHSX2PKZK4JIJJDRDXIRCAG2VXQCTUR62NP76GO7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.686754629 (501h53m32.557837664s) [noise=true , address=45.91.116.8:4045 , piece_id=TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.687518043 (501h53m32.558601078s) [noise=true , address=145.79.227.38:4978 , piece_id=JLOJJ5K6MLTKDKT2EBDZGWAY6HJQCID4UBNHZAOVNXELP5YVHU7A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.693524029 (501h53m32.564607064s) [noise=true , address=145.79.227.45:4231 , piece_id=XZIGODMIDD5BQFXINLEUHZEON6BQANHSLIETFV55UC4NVMD7E5GA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694151704 (501h53m32.565234739s) [noise=true , address=145.79.227.37:4567 , piece_id=LTIMLA6I4ZJLYOTUGGROMJ7N3IAQ7XYLDFCUMJWPJ4UHPK32GLQQ, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694519495 (501h53m32.56560253s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.694951247 (501h53m32.566034282s) [noise=true , address=145.79.227.39:4015 , piece_id=DI2J3QCAO55CGHNSEHVMXU46I4P77NZNGX5MW3I2DN6ISYJ4JT3A, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.695331851 (501h53m32.566414896s) [noise=true , address=145.79.227.6:4258 , piece_id=HKLVNVOO3MIZ6KUNAHQGMRQSGFA37TKA54N34HAYRFRUGWPKCQ6Q, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.695741902 (501h53m32.566824948s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:32.696147946 (501h53m32.567230991s) [noise=true , address=45.91.116.6:4976 , piece_id=PYPVPYVEI6D22PVBBB37EMWUEXM3YKJ5BTIL7MAMXUAX5AL2XVPA, seg_pos={0 0}, upload=440470]: Uploading piece... Dec 26 13:22:34.406954778 (501h53m34.278037813s) [upload=440470]: Next returned nil segment Dec 26 13:22:34.406968063 (501h53m34.278051098s) [upload=440470]: Waiting for error group managing segments... Dec 26 13:22:34.696844389 (501h53m34.567927424s) [noise=true , address=145.79.227.28:4727 , piece_id=SA3AYCPVMMTMD5GQKBNSRSVSDBN6APHPLRSVUTLIKA3YAJFCIJHA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.704462957 (501h53m34.575546002s) [noise=true , address=145.79.227.38:4978 , piece_id=JLOJJ5K6MLTKDKT2EBDZGWAY6HJQCID4UBNHZAOVNXELP5YVHU7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.706884898 (501h53m34.577967943s) [noise=true , address=145.79.227.13:4802 , piece_id=PVACSPBULCNEYH3DED2VIIPVB4FPRBBEYVNBEH4VQWHH3BPNWGBA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.710527174 (501h53m34.581610219s) [noise=true , address=145.79.227.6:4258 , piece_id=HKLVNVOO3MIZ6KUNAHQGMRQSGFA37TKA54N34HAYRFRUGWPKCQ6Q, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.716355412 (501h53m34.587438487s) [noise=true , address=145.79.227.45:4231 , piece_id=XZIGODMIDD5BQFXINLEUHZEON6BQANHSLIETFV55UC4NVMD7E5GA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.722119418 (501h53m34.593202463s) [noise=true , address=145.79.227.20:4438 , piece_id=ASLXFWRJUW4BHSX2PKZK4JIJJDRDXIRCAG2VXQCTUR62NP76GO7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.722645529 (501h53m34.593728575s) [noise=true , address=145.79.227.19:4078 , piece_id=XIGHVBQOBXXURFLGSQC54WCV3PT6IS2VKNBX3IIMDNFF7UM2UT7Q, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.731926392 (501h53m34.603009437s) [noise=true , address=145.79.227.37:4567 , piece_id=LTIMLA6I4ZJLYOTUGGROMJ7N3IAQ7XYLDFCUMJWPJ4UHPK32GLQQ, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.741636291 (501h53m34.612719336s) [noise=true , address=145.79.227.39:4015 , piece_id=DI2J3QCAO55CGHNSEHVMXU46I4P77NZNGX5MW3I2DN6ISYJ4JT3A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:34.966402947 (501h53m34.837485992s) [noise=true , address=45.91.116.6:4976 , piece_id=PYPVPYVEI6D22PVBBB37EMWUEXM3YKJ5BTIL7MAMXUAX5AL2XVPA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.200199657 (501h53m35.071282702s) [noise=true , address=77.90.31.13:4627 , piece_id=SHXBCOH4RVCH3LTMDY6T2FOXQUY42MFFQL6QUR7N6LZL65PF56NA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.482129723 (501h53m35.353212758s) [noise=true , address=45.91.116.8:4045 , piece_id=TRFFV36QG2DVYTHRASMM7BHSWWL65N2QVRSWW7RUZ5UD2ZPOMQSA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.673121025 (501h53m35.54420406s) [noise=true , address=77.90.31.16:4735 , piece_id=ABGDC6L3QWBHLA223Y3CGCO25ZE5ROISP5HYH4QDGXOEWONBPY7A, seg_pos={0 0}, upload=440470]: Done uploading piece. err: <nil> Dec 26 13:22:35.673140882 (501h53m35.544223917s) [seg_pos={0 0}, upload=440470]: Segment reached optimal threshold of 13 pieces. Dec 26 13:22:35.673324491 (501h53m35.544407536s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673332608 (501h53m35.544415653s) [noise=true , address=45.91.116.5:4299 , piece_id=GWH2PWVIQGYKCXPOLCQ2FPWAWEK7TKAXQY5ZN7DC242ZGENLG3YA, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673511408 (501h53m35.544594443s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673517529 (501h53m35.544600574s) [noise=true , address=45.91.116.10:4084 , piece_id=WLGTNJTDEYLH4JNTOTY4P7OAJ3UBPH5OHEVR7YB7DAT76S3C3RWA, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673559619 (501h53m35.544642654s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:35.673580289 (501h53m35.544663334s) [noise=true , address=45.91.116.9:4261 , piece_id=P5C6FXTWSWCTMVLVZV3PJQQPYZBZIHSHKTPEL7VNRL4IZU7SH3LQ, seg_pos={0 0}, upload=440470]: Long tail cancellation due to optimal threshold Dec 26 13:22:35.673627980 (501h53m35.544711025s) [seg_pos={0 0}, upload=440470]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:22:35.673776222 (501h53m35.544859277s) [upload=440470]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:22:35.673784368 (501h53m35.544867413s) [upload=440470]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:22:35.965152704 (501h53m35.836235749s) [upload=440470]: Done uploading segments... Dec 26 13:22:35.965174246 (501h53m35.836257291s) [upload=440470]: Upload finished. err: <nil> Duration: 30.004483617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............X.XXX...X..X...................XX.X........X.XX] duration: 1m0.003164663s trace: https://jaeger.dev.storj.io/trace/7d5c5a1b0fb75195 FAIL (2025-12-26 13:22:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, Piece ID: EREAMZF2HEDF22RSOFKHXZD4PKUYIZ4A4EGLYVCC3ITTKRMBPISQ) context deadline exceeded; piecestore: (Node ID: 12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, Piece ID: 2XKMY5GGXMYEKOCWZPCW5IK2EUK2EMR22EZS5ZUOGBGZBNV5CUVA) context deadline exceeded; piecestore: (Node ID: 12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, Piece ID: LLXFQILXY4DYDECS2EP2MDVAIUX3L3DDICDTLV7TBHUFPM6U3IRA) context deadline exceeded ======LOGS====== rue , address=77.90.31.11:4557 , piece_id=J3DDGITALRWSKTGOYJRMDVRZLYUWIAJE7DF3QLUUPMRLJLNF7NTA, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.122104643 (501h52m57.993187688s) [noise=true , address=77.90.31.10:4548 , piece_id=VNFYUINJGKOQ54QASLQEGZHGTSG6ETR7LIBOT3GW7QKVTQH3AZ7A, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.122570680 (501h52m57.993653725s) [noise=true , address=77.90.31.21:4901 , piece_id=LLXFQILXY4DYDECS2EP2MDVAIUX3L3DDICDTLV7TBHUFPM6U3IRA, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.123172475 (501h52m57.99425551s) [noise=true , address=145.79.227.48:4329 , piece_id=QMODGACPCTY4G4EVWD5TVANPBWRCX5GBHILIOOMZJXPTDK23QPSQ, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.123629856 (501h52m57.994712901s) [noise=true , address=145.79.227.8:4356 , piece_id=BFAAVWE7GQGCNSP3OIVK2SLJXFVX7WGG6G7TDXRVX5G4QCTPIRRA, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.124105813 (501h52m57.995188858s) [noise=true , address=64.49.13.169:4710 , piece_id=QANYMPPKL5P4LYOWGEVMWCJJLPLZEJ3NWDD5RINRTOVGYRSPJONQ, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.124613861 (501h52m57.995696906s) [noise=true , address=77.90.31.12:4602 , piece_id=MKPJGN4POGMMMKGAI34PU6KTL7FVY35PJRU44PTLJ3QD2OWDDPAQ, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.125102822 (501h52m57.996185857s) [noise=true , address=145.79.227.44:4197 , piece_id=42RM5SPCGQCX4HL4JLH36JZXIEBO2MRKWF3IN4MUDE5F6O3JBRWQ, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.127237797 (501h52m57.998320832s) [noise=true , address=45.91.116.9:4267 , piece_id=EREAMZF2HEDF22RSOFKHXZD4PKUYIZ4A4EGLYVCC3ITTKRMBPISQ, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.129228627 (501h52m58.000311672s) [noise=true , address=77.90.31.15:4701 , piece_id=MTNZXXBPY25WJOVLSR3CFRFAKKTJ2GGN6CR3G7PW7D7TYTRZIAVA, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.133899292 (501h52m58.004982337s) [noise=true , address=45.91.116.8:4062 , piece_id=SYCIFRJ3TUN4TKBWGVZWNZRGXSTI73VVE4Q4BF5H3C2L3HEL6WJQ, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.137722962 (501h52m58.008806007s) [noise=true , address=77.90.31.17:4769 , piece_id=2XKMY5GGXMYEKOCWZPCW5IK2EUK2EMR22EZS5ZUOGBGZBNV5CUVA, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.139774939 (501h52m58.010857984s) [noise=true , address=145.79.227.43:4180 , piece_id=NCAOQOVDJLNOKF4P3KFP7QUJG2ETRVLYSDTT5RU5NMU6SPJPQCPA, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.142597925 (501h52m58.01368097s) [noise=true , address=64.49.13.169:4715 , piece_id=GWJUBM6CG33BBD6HOG2YD4NUE6AAAI44C7EL5QFKE4BRJ2SL3ATQ, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:58.145142862 (501h52m58.016225907s) [noise=true , address=145.79.227.40:4079 , piece_id=XDV7O5BTQUNEWZOHQ7WLOADXBPWZIJH4XH4R5ORGF53562OOTBKQ, seg_pos={0 0}, upload=440459]: Uploading piece... Dec 26 13:21:59.611609694 (501h52m59.482692729s) [upload=440459]: Next returned nil segment Dec 26 13:21:59.611626295 (501h52m59.48270934s) [upload=440459]: Waiting for error group managing segments... Dec 26 13:22:00.033116358 (501h52m59.904199393s) [noise=true , address=145.79.227.48:4329 , piece_id=QMODGACPCTY4G4EVWD5TVANPBWRCX5GBHILIOOMZJXPTDK23QPSQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.034182087 (501h52m59.905265122s) [noise=true , address=145.79.227.8:4356 , piece_id=BFAAVWE7GQGCNSP3OIVK2SLJXFVX7WGG6G7TDXRVX5G4QCTPIRRA, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.221448206 (501h53m0.092531251s) [noise=true , address=77.90.31.21:4901 , piece_id=LLXFQILXY4DYDECS2EP2MDVAIUX3L3DDICDTLV7TBHUFPM6U3IRA, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.278959207 (501h53m0.150042242s) [noise=true , address=77.90.31.17:4769 , piece_id=2XKMY5GGXMYEKOCWZPCW5IK2EUK2EMR22EZS5ZUOGBGZBNV5CUVA, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.280867701 (501h53m0.151950736s) [noise=true , address=77.90.31.10:4548 , piece_id=VNFYUINJGKOQ54QASLQEGZHGTSG6ETR7LIBOT3GW7QKVTQH3AZ7A, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.282731349 (501h53m0.153814394s) [noise=true , address=45.91.116.9:4267 , piece_id=EREAMZF2HEDF22RSOFKHXZD4PKUYIZ4A4EGLYVCC3ITTKRMBPISQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.310748720 (501h53m0.181831755s) [noise=true , address=45.91.116.8:4062 , piece_id=SYCIFRJ3TUN4TKBWGVZWNZRGXSTI73VVE4Q4BF5H3C2L3HEL6WJQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.415276168 (501h53m0.286359204s) [noise=true , address=145.79.227.43:4180 , piece_id=NCAOQOVDJLNOKF4P3KFP7QUJG2ETRVLYSDTT5RU5NMU6SPJPQCPA, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.902314417 (501h53m0.773397452s) [noise=true , address=145.79.227.44:4197 , piece_id=42RM5SPCGQCX4HL4JLH36JZXIEBO2MRKWF3IN4MUDE5F6O3JBRWQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:00.995913355 (501h53m0.86699639s) [noise=true , address=145.79.227.34:4929 , piece_id=2DPVMMFFDPCFG5LA2VIH3DPSQMAT42L25FREFKACZDCHLT4STFDA, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:01.152796528 (501h53m1.023879573s) [noise=true , address=145.79.227.40:4079 , piece_id=XDV7O5BTQUNEWZOHQ7WLOADXBPWZIJH4XH4R5ORGF53562OOTBKQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:01.164341923 (501h53m1.035424958s) [noise=true , address=77.90.31.11:4557 , piece_id=J3DDGITALRWSKTGOYJRMDVRZLYUWIAJE7DF3QLUUPMRLJLNF7NTA, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:01.252438305 (501h53m1.12352134s) [noise=true , address=77.90.31.12:4602 , piece_id=MKPJGN4POGMMMKGAI34PU6KTL7FVY35PJRU44PTLJ3QD2OWDDPAQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:01.264018615 (501h53m1.13510166s) [noise=true , address=45.91.116.12:4233 , piece_id=YW73SZOA27EDSMAXJVMDW4LHOFYOKOUNY5X5CLRZ4CXFDKGU4NBA, seg_pos={0 0}, upload=440459]: Done uploading piece. err: <nil> Dec 26 13:22:01.264045607 (501h53m1.135128652s) [seg_pos={0 0}, upload=440459]: Segment reached optimal threshold of 14 pieces. Dec 26 13:22:01.264311514 (501h53m1.135394559s) [noise=true , address=45.91.116.5:4327 , piece_id=R75ZK4DHJQZSUZTKXDA2TJ7NOIK45FEH7N54UNBIOLUNSN56K6KQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:01.264320520 (501h53m1.135403555s) [noise=true , address=45.91.116.5:4327 , piece_id=R75ZK4DHJQZSUZTKXDA2TJ7NOIK45FEH7N54UNBIOLUNSN56K6KQ, seg_pos={0 0}, upload=440459]: Long tail cancellation due to optimal threshold Dec 26 13:22:01.264698821 (501h53m1.135781866s) [noise=true , address=64.49.13.169:4710 , piece_id=QANYMPPKL5P4LYOWGEVMWCJJLPLZEJ3NWDD5RINRTOVGYRSPJONQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:42410->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:42410->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42410->64.49.13.169:4710: use of closed network connection Dec 26 13:22:01.264708659 (501h53m1.135791694s) [noise=true , address=64.49.13.169:4710 , piece_id=QANYMPPKL5P4LYOWGEVMWCJJLPLZEJ3NWDD5RINRTOVGYRSPJONQ, seg_pos={0 0}, upload=440459]: Long tail cancellation due to optimal threshold Dec 26 13:22:01.264830732 (501h53m1.135913777s) [noise=true , address=64.49.13.169:4715 , piece_id=GWJUBM6CG33BBD6HOG2YD4NUE6AAAI44C7EL5QFKE4BRJ2SL3ATQ, seg_pos={0 0}, upload=440459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:34318->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:34318->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34318->64.49.13.169:4715: use of closed network connection Dec 26 13:22:01.264839058 (501h53m1.135922103s) [noise=true , address=64.49.13.169:4715 , piece_id=GWJUBM6CG33BBD6HOG2YD4NUE6AAAI44C7EL5QFKE4BRJ2SL3ATQ, seg_pos={0 0}, upload=440459]: Long tail cancellation due to optimal threshold Dec 26 13:22:01.265019081 (501h53m1.136102126s) [noise=true , address=77.90.31.15:4701 , piece_id=MTNZXXBPY25WJOVLSR3CFRFAKKTJ2GGN6CR3G7PW7D7TYTRZIAVA, seg_pos={0 0}, upload=440459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 26 13:22:01.265029119 (501h53m1.136112164s) [noise=true , address=77.90.31.15:4701 , piece_id=MTNZXXBPY25WJOVLSR3CFRFAKKTJ2GGN6CR3G7PW7D7TYTRZIAVA, seg_pos={0 0}, upload=440459]: Long tail cancellation due to optimal threshold Dec 26 13:22:01.265053085 (501h53m1.13613613s) [seg_pos={0 0}, upload=440459]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:22:01.265126505 (501h53m1.13620955s) [upload=440459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:22:01.265131464 (501h53m1.136214499s) [upload=440459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:22:01.559750883 (501h53m1.430833938s) [upload=440459]: Done uploading segments... Dec 26 13:22:01.559781470 (501h53m1.430864506s) [upload=440459]: Upload finished. err: <nil> Duration: 1m0.003188069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [...XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX] duration: 5.001440586s trace: https://jaeger.dev.storj.io/trace/cde8d6d2d0dcc52 FAIL (2025-12-26 13:22:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001464621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [....XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XX] duration: 5.000752286s trace: https://jaeger.dev.storj.io/trace/10db0424157b32ad FAIL (2025-12-26 13:22:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0007656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.] duration: 6.074819471s trace: https://jaeger.dev.storj.io/trace/3b02b4ca2747dfdf FAIL (2025-12-26 13:22:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 26 13:21:41.998133328 (501h52m41.869216394s) [upload=440454]: Got next segment. Inline: false Dec 26 13:21:41.998178746 (501h52m41.869261781s) [upload=440454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:21:41.998187492 (501h52m41.869270527s) [upload=440454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:21:42.246504943 (501h52m42.117587988s) [seg_pos={0 0}, upload=440454]: Begin upload segment... Dec 26 13:21:42.246553655 (501h52m42.1176367s) [seg_pos={0 0}, upload=440454]: Done begin upload segment. Dec 26 13:21:42.246606947 (501h52m42.117689992s) [noise=true , address=45.91.116.8:4048 , piece_id=SJRX5WQYSPWNFYMTLDTR3XGAP5MPKK4RWAKVHWI4KKCSM3UZMC3Q, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.246623568 (501h52m42.117706614s) [noise=true , address=45.91.116.9:4261 , piece_id=E5NTA4U5TWBKJUGDVJSFQX5UFOZFPXTLSI7JHFDERYV42SV6AMYA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.247351915 (501h52m42.118434961s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYK2RKT4QKZBZS36IVSA7V73OL52XOHK566AFEBCKPFDDFIHSSZA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.247566043 (501h52m42.118649088s) [noise=true , address=145.79.227.42:4123 , piece_id=NEAIRHXZ7OHWA6MK6WQ7WKYR6RCKQIHGNFHYS3CTUOSJWYWCVVDQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.247642249 (501h52m42.118725284s) [noise=true , address=145.79.227.12:4765 , piece_id=2RUY7MYRLG4TVFGAMD736MVBXAMQUMI73R6E2DAOOLVWSMHYQVWQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.247922302 (501h52m42.119005347s) [noise=true , address=45.91.116.11:4118 , piece_id=H5PRUPLHTB4GO7ZOFJHKUF2OZXLZG2JVIPO6ZEPDYY2BZSY5BOMQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248035137 (501h52m42.119118172s) [noise=true , address=145.79.227.17:4970 , piece_id=3ISZV6AZZJ7OLNAGLNZ2JHWK4E46CAQHVMSOGYV3TT2I4Q64CKWQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248235278 (501h52m42.119318313s) [noise=true , address=145.79.227.22:4511 , piece_id=AS3JVZWYDR7GUWNUD6W7QMAIZZMJ4NTZ33M5YL73V3XS56XXNP5A, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248477209 (501h52m42.119560254s) [noise=true , address=45.91.116.10:4082 , piece_id=U25M5IWQ7ZNGNO5J4OJPEKIHEEHTHH3VIIMJP36LQ4LRLWCZVIDA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248581928 (501h52m42.119664963s) [noise=true , address=145.79.227.15:4874 , piece_id=56S2H5V46RSDWUGP6J545SH6AFWQDWW7GXTZL26BXA7YCVJY6GWA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248887900 (501h52m42.119970945s) [noise=true , address=45.91.116.6:4975 , piece_id=6BAH5CXPNIRTED7MWIRDGG2VQQ6DKMMY4YL2WUQNBOZBE2URRKXA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248895125 (501h52m42.11997817s) [noise=true , address=145.79.227.4:4186 , piece_id=3TWQUQDGFS7BHFD4D2MVB24GUVUZAO4WK3IRSW6KC3CHTYDDJESQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.249287682 (501h52m42.120370727s) [noise=true , address=145.79.227.6:4258 , piece_id=KBKD2OC4TXVFVDYW2WLDTVXZY2P2ZNDXRHKCWJQKYHLTSX44XGHA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.249602782 (501h52m42.120685827s) [noise=true , address=145.79.227.2:4113 , piece_id=6FZ37VXCTOCMZQ3LHAVNSOG6QVHCHJASCZFDM273ANYSVJRH2IBA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.250866017 (501h52m42.121949052s) [noise=true , address=145.79.227.31:4799 , piece_id=K76W7OS7ZZIGDYGV77PTJFPTZSVNHQ22GWQ5JCUBWT2EHRLFYTKA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.251530813 (501h52m42.122613848s) [noise=true , address=145.79.227.19:4078 , piece_id=SBVIAKSWQU7TFNFJQUUXXMHUOFZ2R24MQSDPMS6L7BV54PD2BFTQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:43.960323139 (501h52m43.831406174s) [upload=440454]: Next returned nil segment Dec 26 13:21:43.960337777 (501h52m43.831420822s) [upload=440454]: Waiting for error group managing segments... Dec 26 13:21:44.248467497 (501h52m44.119550532s) [noise=true , address=145.79.227.17:4970 , piece_id=3ISZV6AZZJ7OLNAGLNZ2JHWK4E46CAQHVMSOGYV3TT2I4Q64CKWQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.249139447 (501h52m44.120222492s) [noise=true , address=145.79.227.22:4511 , piece_id=AS3JVZWYDR7GUWNUD6W7QMAIZZMJ4NTZ33M5YL73V3XS56XXNP5A, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.249896478 (501h52m44.120979523s) [noise=true , address=145.79.227.31:4799 , piece_id=K76W7OS7ZZIGDYGV77PTJFPTZSVNHQ22GWQ5JCUBWT2EHRLFYTKA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.252476311 (501h52m44.123559356s) [noise=true , address=145.79.227.6:4258 , piece_id=KBKD2OC4TXVFVDYW2WLDTVXZY2P2ZNDXRHKCWJQKYHLTSX44XGHA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.254259135 (501h52m44.12534217s) [noise=true , address=145.79.227.12:4765 , piece_id=2RUY7MYRLG4TVFGAMD736MVBXAMQUMI73R6E2DAOOLVWSMHYQVWQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.254631144 (501h52m44.125714179s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYK2RKT4QKZBZS36IVSA7V73OL52XOHK566AFEBCKPFDDFIHSSZA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.256105000 (501h52m44.127188035s) [noise=true , address=145.79.227.15:4874 , piece_id=56S2H5V46RSDWUGP6J545SH6AFWQDWW7GXTZL26BXA7YCVJY6GWA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.271001836 (501h52m44.142084881s) [noise=true , address=145.79.227.2:4113 , piece_id=6FZ37VXCTOCMZQ3LHAVNSOG6QVHCHJASCZFDM273ANYSVJRH2IBA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.278727778 (501h52m44.149810813s) [noise=true , address=145.79.227.19:4078 , piece_id=SBVIAKSWQU7TFNFJQUUXXMHUOFZ2R24MQSDPMS6L7BV54PD2BFTQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.280968886 (501h52m44.152051921s) [noise=true , address=145.79.227.4:4186 , piece_id=3TWQUQDGFS7BHFD4D2MVB24GUVUZAO4WK3IRSW6KC3CHTYDDJESQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.287991077 (501h52m44.159074112s) [noise=true , address=145.79.227.42:4123 , piece_id=NEAIRHXZ7OHWA6MK6WQ7WKYR6RCKQIHGNFHYS3CTUOSJWYWCVVDQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.559168798 (501h52m44.430251833s) [noise=true , address=45.91.116.10:4082 , piece_id=U25M5IWQ7ZNGNO5J4OJPEKIHEEHTHH3VIIMJP36LQ4LRLWCZVIDA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.624034494 (501h52m44.495117529s) [noise=true , address=45.91.116.11:4118 , piece_id=H5PRUPLHTB4GO7ZOFJHKUF2OZXLZG2JVIPO6ZEPDYY2BZSY5BOMQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.624055845 (501h52m44.49513888s) [seg_pos={0 0}, upload=440454]: Segment reached optimal threshold of 13 pieces. Dec 26 13:21:44.624236258 (501h52m44.495319293s) [noise=true , address=45.91.116.6:4975 , piece_id=6BAH5CXPNIRTED7MWIRDGG2VQQ6DKMMY4YL2WUQNBOZBE2URRKXA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:57474->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:57474->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57474->45.91.116.6:4975: use of closed network connection Dec 26 13:21:44.624242049 (501h52m44.495325084s) [noise=true , address=45.91.116.6:4975 , piece_id=6BAH5CXPNIRTED7MWIRDGG2VQQ6DKMMY4YL2WUQNBOZBE2URRKXA, seg_pos={0 0}, upload=440454]: Long tail cancellation due to optimal threshold Dec 26 13:21:44.624352229 (501h52m44.495435264s) [noise=true , address=45.91.116.8:4048 , piece_id=SJRX5WQYSPWNFYMTLDTR3XGAP5MPKK4RWAKVHWI4KKCSM3UZMC3Q, seg_pos={0 0}, upload=440454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:52520->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:52520->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52520->45.91.116.8:4048: use of closed network connection Dec 26 13:21:44.624357729 (501h52m44.495440764s) [noise=true , address=45.91.116.8:4048 , piece_id=SJRX5WQYSPWNFYMTLDTR3XGAP5MPKK4RWAKVHWI4KKCSM3UZMC3Q, seg_pos={0 0}, upload=440454]: Long tail cancellation due to optimal threshold Dec 26 13:21:44.624502925 (501h52m44.495585971s) [noise=true , address=45.91.116.9:4261 , piece_id=E5NTA4U5TWBKJUGDVJSFQX5UFOZFPXTLSI7JHFDERYV42SV6AMYA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 13:21:44.624523645 (501h52m44.49560668s) [noise=true , address=45.91.116.9:4261 , piece_id=E5NTA4U5TWBKJUGDVJSFQX5UFOZFPXTLSI7JHFDERYV42SV6AMYA, seg_pos={0 0}, upload=440454]: Long tail cancellation due to optimal threshold Dec 26 13:21:44.624602645 (501h52m44.495685701s) [seg_pos={0 0}, upload=440454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:21:44.624770646 (501h52m44.495853701s) [upload=440454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:21:44.624778360 (501h52m44.495861395s) [upload=440454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:21:44.929563416 (501h52m44.800646461s) [upload=440454]: Done uploading segments... Dec 26 13:21:44.929586499 (501h52m44.800669544s) [upload=440454]: Upload finished. err: <nil> Duration: 6.074836895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X] duration: 30.003397219s trace: https://jaeger.dev.storj.io/trace/3b02b4ca2747dfdf FAIL (2025-12-26 13:22:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 26 13:21:41.998133328 (501h52m41.869216394s) [upload=440454]: Got next segment. Inline: false Dec 26 13:21:41.998178746 (501h52m41.869261781s) [upload=440454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:21:41.998187492 (501h52m41.869270527s) [upload=440454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:21:42.246504943 (501h52m42.117587988s) [seg_pos={0 0}, upload=440454]: Begin upload segment... Dec 26 13:21:42.246553655 (501h52m42.1176367s) [seg_pos={0 0}, upload=440454]: Done begin upload segment. Dec 26 13:21:42.246606947 (501h52m42.117689992s) [noise=true , address=45.91.116.8:4048 , piece_id=SJRX5WQYSPWNFYMTLDTR3XGAP5MPKK4RWAKVHWI4KKCSM3UZMC3Q, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.246623568 (501h52m42.117706614s) [noise=true , address=45.91.116.9:4261 , piece_id=E5NTA4U5TWBKJUGDVJSFQX5UFOZFPXTLSI7JHFDERYV42SV6AMYA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.247351915 (501h52m42.118434961s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYK2RKT4QKZBZS36IVSA7V73OL52XOHK566AFEBCKPFDDFIHSSZA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.247566043 (501h52m42.118649088s) [noise=true , address=145.79.227.42:4123 , piece_id=NEAIRHXZ7OHWA6MK6WQ7WKYR6RCKQIHGNFHYS3CTUOSJWYWCVVDQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.247642249 (501h52m42.118725284s) [noise=true , address=145.79.227.12:4765 , piece_id=2RUY7MYRLG4TVFGAMD736MVBXAMQUMI73R6E2DAOOLVWSMHYQVWQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.247922302 (501h52m42.119005347s) [noise=true , address=45.91.116.11:4118 , piece_id=H5PRUPLHTB4GO7ZOFJHKUF2OZXLZG2JVIPO6ZEPDYY2BZSY5BOMQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248035137 (501h52m42.119118172s) [noise=true , address=145.79.227.17:4970 , piece_id=3ISZV6AZZJ7OLNAGLNZ2JHWK4E46CAQHVMSOGYV3TT2I4Q64CKWQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248235278 (501h52m42.119318313s) [noise=true , address=145.79.227.22:4511 , piece_id=AS3JVZWYDR7GUWNUD6W7QMAIZZMJ4NTZ33M5YL73V3XS56XXNP5A, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248477209 (501h52m42.119560254s) [noise=true , address=45.91.116.10:4082 , piece_id=U25M5IWQ7ZNGNO5J4OJPEKIHEEHTHH3VIIMJP36LQ4LRLWCZVIDA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248581928 (501h52m42.119664963s) [noise=true , address=145.79.227.15:4874 , piece_id=56S2H5V46RSDWUGP6J545SH6AFWQDWW7GXTZL26BXA7YCVJY6GWA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248887900 (501h52m42.119970945s) [noise=true , address=45.91.116.6:4975 , piece_id=6BAH5CXPNIRTED7MWIRDGG2VQQ6DKMMY4YL2WUQNBOZBE2URRKXA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.248895125 (501h52m42.11997817s) [noise=true , address=145.79.227.4:4186 , piece_id=3TWQUQDGFS7BHFD4D2MVB24GUVUZAO4WK3IRSW6KC3CHTYDDJESQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.249287682 (501h52m42.120370727s) [noise=true , address=145.79.227.6:4258 , piece_id=KBKD2OC4TXVFVDYW2WLDTVXZY2P2ZNDXRHKCWJQKYHLTSX44XGHA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.249602782 (501h52m42.120685827s) [noise=true , address=145.79.227.2:4113 , piece_id=6FZ37VXCTOCMZQ3LHAVNSOG6QVHCHJASCZFDM273ANYSVJRH2IBA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.250866017 (501h52m42.121949052s) [noise=true , address=145.79.227.31:4799 , piece_id=K76W7OS7ZZIGDYGV77PTJFPTZSVNHQ22GWQ5JCUBWT2EHRLFYTKA, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:42.251530813 (501h52m42.122613848s) [noise=true , address=145.79.227.19:4078 , piece_id=SBVIAKSWQU7TFNFJQUUXXMHUOFZ2R24MQSDPMS6L7BV54PD2BFTQ, seg_pos={0 0}, upload=440454]: Uploading piece... Dec 26 13:21:43.960323139 (501h52m43.831406174s) [upload=440454]: Next returned nil segment Dec 26 13:21:43.960337777 (501h52m43.831420822s) [upload=440454]: Waiting for error group managing segments... Dec 26 13:21:44.248467497 (501h52m44.119550532s) [noise=true , address=145.79.227.17:4970 , piece_id=3ISZV6AZZJ7OLNAGLNZ2JHWK4E46CAQHVMSOGYV3TT2I4Q64CKWQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.249139447 (501h52m44.120222492s) [noise=true , address=145.79.227.22:4511 , piece_id=AS3JVZWYDR7GUWNUD6W7QMAIZZMJ4NTZ33M5YL73V3XS56XXNP5A, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.249896478 (501h52m44.120979523s) [noise=true , address=145.79.227.31:4799 , piece_id=K76W7OS7ZZIGDYGV77PTJFPTZSVNHQ22GWQ5JCUBWT2EHRLFYTKA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.252476311 (501h52m44.123559356s) [noise=true , address=145.79.227.6:4258 , piece_id=KBKD2OC4TXVFVDYW2WLDTVXZY2P2ZNDXRHKCWJQKYHLTSX44XGHA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.254259135 (501h52m44.12534217s) [noise=true , address=145.79.227.12:4765 , piece_id=2RUY7MYRLG4TVFGAMD736MVBXAMQUMI73R6E2DAOOLVWSMHYQVWQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.254631144 (501h52m44.125714179s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYK2RKT4QKZBZS36IVSA7V73OL52XOHK566AFEBCKPFDDFIHSSZA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.256105000 (501h52m44.127188035s) [noise=true , address=145.79.227.15:4874 , piece_id=56S2H5V46RSDWUGP6J545SH6AFWQDWW7GXTZL26BXA7YCVJY6GWA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.271001836 (501h52m44.142084881s) [noise=true , address=145.79.227.2:4113 , piece_id=6FZ37VXCTOCMZQ3LHAVNSOG6QVHCHJASCZFDM273ANYSVJRH2IBA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.278727778 (501h52m44.149810813s) [noise=true , address=145.79.227.19:4078 , piece_id=SBVIAKSWQU7TFNFJQUUXXMHUOFZ2R24MQSDPMS6L7BV54PD2BFTQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.280968886 (501h52m44.152051921s) [noise=true , address=145.79.227.4:4186 , piece_id=3TWQUQDGFS7BHFD4D2MVB24GUVUZAO4WK3IRSW6KC3CHTYDDJESQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.287991077 (501h52m44.159074112s) [noise=true , address=145.79.227.42:4123 , piece_id=NEAIRHXZ7OHWA6MK6WQ7WKYR6RCKQIHGNFHYS3CTUOSJWYWCVVDQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.559168798 (501h52m44.430251833s) [noise=true , address=45.91.116.10:4082 , piece_id=U25M5IWQ7ZNGNO5J4OJPEKIHEEHTHH3VIIMJP36LQ4LRLWCZVIDA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.624034494 (501h52m44.495117529s) [noise=true , address=45.91.116.11:4118 , piece_id=H5PRUPLHTB4GO7ZOFJHKUF2OZXLZG2JVIPO6ZEPDYY2BZSY5BOMQ, seg_pos={0 0}, upload=440454]: Done uploading piece. err: <nil> Dec 26 13:21:44.624055845 (501h52m44.49513888s) [seg_pos={0 0}, upload=440454]: Segment reached optimal threshold of 13 pieces. Dec 26 13:21:44.624236258 (501h52m44.495319293s) [noise=true , address=45.91.116.6:4975 , piece_id=6BAH5CXPNIRTED7MWIRDGG2VQQ6DKMMY4YL2WUQNBOZBE2URRKXA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:57474->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:57474->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57474->45.91.116.6:4975: use of closed network connection Dec 26 13:21:44.624242049 (501h52m44.495325084s) [noise=true , address=45.91.116.6:4975 , piece_id=6BAH5CXPNIRTED7MWIRDGG2VQQ6DKMMY4YL2WUQNBOZBE2URRKXA, seg_pos={0 0}, upload=440454]: Long tail cancellation due to optimal threshold Dec 26 13:21:44.624352229 (501h52m44.495435264s) [noise=true , address=45.91.116.8:4048 , piece_id=SJRX5WQYSPWNFYMTLDTR3XGAP5MPKK4RWAKVHWI4KKCSM3UZMC3Q, seg_pos={0 0}, upload=440454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:52520->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:52520->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52520->45.91.116.8:4048: use of closed network connection Dec 26 13:21:44.624357729 (501h52m44.495440764s) [noise=true , address=45.91.116.8:4048 , piece_id=SJRX5WQYSPWNFYMTLDTR3XGAP5MPKK4RWAKVHWI4KKCSM3UZMC3Q, seg_pos={0 0}, upload=440454]: Long tail cancellation due to optimal threshold Dec 26 13:21:44.624502925 (501h52m44.495585971s) [noise=true , address=45.91.116.9:4261 , piece_id=E5NTA4U5TWBKJUGDVJSFQX5UFOZFPXTLSI7JHFDERYV42SV6AMYA, seg_pos={0 0}, upload=440454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 13:21:44.624523645 (501h52m44.49560668s) [noise=true , address=45.91.116.9:4261 , piece_id=E5NTA4U5TWBKJUGDVJSFQX5UFOZFPXTLSI7JHFDERYV42SV6AMYA, seg_pos={0 0}, upload=440454]: Long tail cancellation due to optimal threshold Dec 26 13:21:44.624602645 (501h52m44.495685701s) [seg_pos={0 0}, upload=440454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:21:44.624770646 (501h52m44.495853701s) [upload=440454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:21:44.624778360 (501h52m44.495861395s) [upload=440454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:21:44.929563416 (501h52m44.800646461s) [upload=440454]: Done uploading segments... Dec 26 13:21:44.929586499 (501h52m44.800669544s) [upload=440454]: Upload finished. err: <nil> Duration: 30.003422587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............X.XXX...X..X...................XX.X........X.X] duration: 1m0.002889023s trace: https://jaeger.dev.storj.io/trace/30ad4163e0961bc6 FAIL (2025-12-26 13:21:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, Piece ID: 4DBP3SBBZKVMWTNUYNNS5C5QKKH3KKQW7EJWCZHV7LB35KK6GQIA) context deadline exceeded; piecestore: (Node ID: 1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, Piece ID: ZKYWBZIXSSV23SMYOAFOXROTFM6XHZV63KLPGT5JDOPHEQGCKE7A) context deadline exceeded; piecestore: (Node ID: 16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, Piece ID: CTNVX7SFS7VWWVRNIHL6TBAGU4TECE3JPNBW5ZLSS4C4IFPRP6LA) context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.760165126 (501h51m51.631248161s) [noise=true , address=64.49.13.169:4715 , piece_id=7G7BCCNVKELD7NRPCEGOKEQCFAFFX5NZSWBUTI4NC52MZX3TR56Q, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.760617728 (501h51m51.631700773s) [noise=true , address=77.90.31.14:4674 , piece_id=W24ML5HV3ICOKMOPVNAUD6KKIM3XPKRCQ3KYLKPWGGR47AU3IABA, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.761023511 (501h51m51.632106556s) [noise=true , address=77.90.31.13:4651 , piece_id=M6G46GLTM62DQ7FY64WLGQRKP6KVUS4BTFHUUXN23XQJKCW2KPUA, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.761379398 (501h51m51.632462434s) [noise=true , address=45.91.116.8:4051 , piece_id=ZKYWBZIXSSV23SMYOAFOXROTFM6XHZV63KLPGT5JDOPHEQGCKE7A, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.761990391 (501h51m51.633073436s) [noise=true , address=145.79.227.33:4894 , piece_id=5TZEHV6Y54OY2UQT5K2PUPUNXTP4QVK7IGMTJD5EIBZ5H7YIDPGQ, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.762413758 (501h51m51.633496793s) [noise=true , address=145.79.227.34:4911 , piece_id=EH46AQBFOK4BANFLDZLWRL633UPRM3W77JLTRWAFMXJIBYA6DX5A, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.763004453 (501h51m51.634087528s) [noise=true , address=45.91.116.6:4007 , piece_id=CTNVX7SFS7VWWVRNIHL6TBAGU4TECE3JPNBW5ZLSS4C4IFPRP6LA, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.763447957 (501h51m51.634531002s) [noise=true , address=145.79.227.44:4218 , piece_id=HWRIGV6WPG7RUESHGZ7UZKC2IUHB3M6AOKYXGR2XJ6346QLC7TFA, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.763948521 (501h51m51.635031566s) [noise=true , address=145.79.227.40:4052 , piece_id=JRIKQS6DBOWXX7CZWVWTXYCOUTJH2JWARNUOPSWGTGFJXL5XVZWQ, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.764353712 (501h51m51.635436747s) [noise=true , address=145.79.227.49:4341 , piece_id=JRBZSCZVJYBOSZA7SO25CLWC72JSEOJCLY2SJYNKXMMULWLY4W3Q, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.764739145 (501h51m51.635822191s) [noise=true , address=77.90.31.21:4900 , piece_id=2KPRZDRLDYN3YCBPCO25W7QQ6OZVWPKASPNME57PUNIT4MFSYGAQ, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.765273854 (501h51m51.636356889s) [noise=true , address=145.79.227.52:4450 , piece_id=UEQ73X5KHGNPQTYHWIILRW55F6TCJL4W3EE5XWAFCLPNCPAFSY3Q, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.765650111 (501h51m51.636733146s) [noise=true , address=77.90.31.12:4604 , piece_id=4DBP3SBBZKVMWTNUYNNS5C5QKKH3KKQW7EJWCZHV7LB35KK6GQIA, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.766393887 (501h51m51.637476922s) [noise=true , address=64.49.13.138:4491 , piece_id=PGZVCBFTLWGYOO7HV6LFMJ2OUWWYP2LO675DBLCJ5IFDZ7JQXIIA, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:51.766837492 (501h51m51.637920527s) [noise=true , address=145.79.227.30:4796 , piece_id=NY4IS56TB3P63NKMV5NJSAMN2ZRQA7SHDSIDVR5GLT6BAUTHH75A, seg_pos={0 0}, upload=440443]: Uploading piece... Dec 26 13:20:53.076006733 (501h51m52.947089768s) [upload=440443]: Next returned nil segment Dec 26 13:20:53.076024678 (501h51m52.947107723s) [upload=440443]: Waiting for error group managing segments... Dec 26 13:20:53.431887063 (501h51m53.302970098s) [noise=true , address=64.49.13.138:4491 , piece_id=PGZVCBFTLWGYOO7HV6LFMJ2OUWWYP2LO675DBLCJ5IFDZ7JQXIIA, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:53.651008159 (501h51m53.522091224s) [noise=true , address=145.79.227.49:4341 , piece_id=JRBZSCZVJYBOSZA7SO25CLWC72JSEOJCLY2SJYNKXMMULWLY4W3Q, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:53.722199770 (501h51m53.593282815s) [noise=true , address=145.79.227.52:4450 , piece_id=UEQ73X5KHGNPQTYHWIILRW55F6TCJL4W3EE5XWAFCLPNCPAFSY3Q, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:53.838084926 (501h51m53.709167971s) [noise=true , address=77.90.31.14:4674 , piece_id=W24ML5HV3ICOKMOPVNAUD6KKIM3XPKRCQ3KYLKPWGGR47AU3IABA, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:53.853521851 (501h51m53.724604886s) [noise=true , address=145.79.227.34:4911 , piece_id=EH46AQBFOK4BANFLDZLWRL633UPRM3W77JLTRWAFMXJIBYA6DX5A, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:53.912634622 (501h51m53.783717667s) [noise=true , address=77.90.31.11:4578 , piece_id=IWIELWJNLY43MPIYM4EHYILYMHBC64UFLRCPRQUW26NB6MOCPACA, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:53.928690236 (501h51m53.799773271s) [noise=true , address=77.90.31.21:4900 , piece_id=2KPRZDRLDYN3YCBPCO25W7QQ6OZVWPKASPNME57PUNIT4MFSYGAQ, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:53.959145557 (501h51m53.830228612s) [noise=true , address=77.90.31.19:4850 , piece_id=U5ILJKH6ODJVLZEPRH64LVGKTU3QLYU4KL5CRQ5FPIG5CF7DC4BQ, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:54.002856530 (501h51m53.873939566s) [noise=true , address=45.91.116.6:4007 , piece_id=CTNVX7SFS7VWWVRNIHL6TBAGU4TECE3JPNBW5ZLSS4C4IFPRP6LA, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:54.617318837 (501h51m54.488401882s) [noise=true , address=145.79.227.30:4796 , piece_id=NY4IS56TB3P63NKMV5NJSAMN2ZRQA7SHDSIDVR5GLT6BAUTHH75A, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:54.622380305 (501h51m54.49346334s) [noise=true , address=145.79.227.44:4218 , piece_id=HWRIGV6WPG7RUESHGZ7UZKC2IUHB3M6AOKYXGR2XJ6346QLC7TFA, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:54.667673901 (501h51m54.538756947s) [noise=true , address=77.90.31.12:4604 , piece_id=4DBP3SBBZKVMWTNUYNNS5C5QKKH3KKQW7EJWCZHV7LB35KK6GQIA, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:54.826881422 (501h51m54.697964467s) [noise=true , address=77.90.31.13:4651 , piece_id=M6G46GLTM62DQ7FY64WLGQRKP6KVUS4BTFHUUXN23XQJKCW2KPUA, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:54.840354396 (501h51m54.711437441s) [noise=true , address=45.91.116.8:4051 , piece_id=ZKYWBZIXSSV23SMYOAFOXROTFM6XHZV63KLPGT5JDOPHEQGCKE7A, seg_pos={0 0}, upload=440443]: Done uploading piece. err: <nil> Dec 26 13:20:54.840373022 (501h51m54.711456067s) [seg_pos={0 0}, upload=440443]: Segment reached optimal threshold of 14 pieces. Dec 26 13:20:54.840542335 (501h51m54.71162538s) [noise=true , address=64.49.13.169:4715 , piece_id=7G7BCCNVKELD7NRPCEGOKEQCFAFFX5NZSWBUTI4NC52MZX3TR56Q, seg_pos={0 0}, upload=440443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:41326->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:41326->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41326->64.49.13.169:4715: use of closed network connection Dec 26 13:20:54.840548046 (501h51m54.711631091s) [noise=true , address=64.49.13.169:4715 , piece_id=7G7BCCNVKELD7NRPCEGOKEQCFAFFX5NZSWBUTI4NC52MZX3TR56Q, seg_pos={0 0}, upload=440443]: Long tail cancellation due to optimal threshold Dec 26 13:20:54.840839150 (501h51m54.711922185s) [noise=true , address=145.79.227.40:4052 , piece_id=JRIKQS6DBOWXX7CZWVWTXYCOUTJH2JWARNUOPSWGTGFJXL5XVZWQ, seg_pos={0 0}, upload=440443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 26 13:20:54.840847064 (501h51m54.711930109s) [noise=true , address=145.79.227.40:4052 , piece_id=JRIKQS6DBOWXX7CZWVWTXYCOUTJH2JWARNUOPSWGTGFJXL5XVZWQ, seg_pos={0 0}, upload=440443]: Long tail cancellation due to optimal threshold Dec 26 13:20:54.841032398 (501h51m54.712115443s) [noise=true , address=145.79.227.33:4894 , piece_id=5TZEHV6Y54OY2UQT5K2PUPUNXTP4QVK7IGMTJD5EIBZ5H7YIDPGQ, seg_pos={0 0}, upload=440443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: expected piece hash; context canceled; EOF Dec 26 13:20:54.841038249 (501h51m54.712121284s) [noise=true , address=145.79.227.33:4894 , piece_id=5TZEHV6Y54OY2UQT5K2PUPUNXTP4QVK7IGMTJD5EIBZ5H7YIDPGQ, seg_pos={0 0}, upload=440443]: Long tail cancellation due to optimal threshold Dec 26 13:20:54.841364911 (501h51m54.712447956s) [noise=true , address=77.90.31.15:4719 , piece_id=YAA7Q67XORGHN3XBFR36GLLJI4DIWSBJOU3LAF42WXPGXTU75FVQ, seg_pos={0 0}, upload=440443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qyhppjd6m6he7eHVZLuhK9BYKXq1DY3GuKBNfyb97JVaz9wyk): protocol: read tcp 188.166.225.170:51196->77.90.31.15:4719: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:20:54.841372254 (501h51m54.712455289s) [noise=true , address=77.90.31.15:4719 , piece_id=YAA7Q67XORGHN3XBFR36GLLJI4DIWSBJOU3LAF42WXPGXTU75FVQ, seg_pos={0 0}, upload=440443]: Long tail cancellation due to optimal threshold Dec 26 13:20:54.841391311 (501h51m54.712474346s) [seg_pos={0 0}, upload=440443]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:20:54.841472054 (501h51m54.71255509s) [upload=440443]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:20:54.841477575 (501h51m54.71256062s) [upload=440443]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:20:55.147885230 (501h51m55.018968275s) [upload=440443]: Done uploading segments... Dec 26 13:20:55.147926779 (501h51m55.019009814s) [upload=440443]: Upload finished. err: <nil> Duration: 1m0.002902108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.....XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.X] duration: 5.001806844s trace: https://jaeger.dev.storj.io/trace/676c7efc9e80498c FAIL (2025-12-26 13:21:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001831321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [...............X.XXX...X..X...................XX.X........X.] duration: 54.369096901s trace: https://jaeger.dev.storj.io/trace/3764360f5fc66fe4 FAIL (2025-12-26 13:20:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo, Piece ID: HEMHNVQKVNTUDI3YUYUTBFB4KTDU52M2CY5CS6HLDNISY4YO4Q2A) context deadline exceeded; piecestore: (Node ID: 1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, Piece ID: JCL5I57XK4XD6VYK4USM6HA7VI4HMRQ3455PYCJUGS267H5UD5KA) context deadline exceeded; piecestore: (Node ID: 12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, Piece ID: K37R3G6TOSNYRHQVD7TCRNXFKLFZ542EWYOLFXHSMEL7UJWZ2GZQ) context deadline exceeded ======LOGS====== .91.116.9:4267 , piece_id=JCL5I57XK4XD6VYK4USM6HA7VI4HMRQ3455PYCJUGS267H5UD5KA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.932725752 (501h49m56.803808797s) [noise=true , address=145.79.227.40:4054 , piece_id=IZFFBTPOADF6BYIYODQVAVWPUWI2KLKTCRPALMPHGU3ZGLUBXCHA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.933157144 (501h49m56.804240179s) [noise=true , address=45.91.116.5:4327 , piece_id=K37R3G6TOSNYRHQVD7TCRNXFKLFZ542EWYOLFXHSMEL7UJWZ2GZQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.933517910 (501h49m56.804600945s) [noise=true , address=45.91.116.13:4167 , piece_id=MADAUEX7VYZP7B2I4QCLGGEELNXKWJIC7IN4MO5DOQUQHT7QCKMQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.933929144 (501h49m56.805012179s) [noise=true , address=77.90.31.10:4552 , piece_id=W4HPIARTL67DK24ONOCXV43GDWO5N72VOT3NDIIK222SOQSEOLWA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.934322603 (501h49m56.805405648s) [noise=true , address=145.79.227.8:4356 , piece_id=T4YWHFUJZNM5GCZBF4BF36GUJTZVSNMW3SMBCILKWB4VCQATHDRQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.934734637 (501h49m56.805817682s) [noise=true , address=193.149.164.143:4702 , piece_id=WCDUGPEQ4OKX5LF6UAKAUI7IIADUGICUWXCKJJZLBQPLR4IQWCMQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.935094532 (501h49m56.806177577s) [noise=true , address=145.79.227.33:4897 , piece_id=YGEFUI4Z26M7DQVOGKB2IPCWCQATVNZAAQ6SDGOUA72XA5BOC2JA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.936684089 (501h49m56.807767124s) [noise=true , address=145.79.227.44:4218 , piece_id=XMGFRL4GQ5IJX4J4BHTA6EOWUJHQ2BJF47KBBKUSKPRFUFG6GVIQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.937029768 (501h49m56.808112813s) [noise=true , address=145.79.227.30:4767 , piece_id=QDGOXWPYO5UCENH6DXG46ONRVDCZORDIV7TDC3FIKYXR4S64Q5MQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.938672836 (501h49m56.809755871s) [noise=true , address=64.49.13.26:4945 , piece_id=AIHRNVIXZXAHLHRKZ4HC5TX5FAUN33C4UWEXID2LU2RNGIQQSOAA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.942723659 (501h49m56.813806704s) [noise=true , address=145.79.227.52:4449 , piece_id=SUWWUSJ4NDCMJNRGZ3NU2WMBLISLHYPXJNFB6ISVTH6LHGK6M7OQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.944292236 (501h49m56.815375271s) [noise=true , address=77.90.31.21:4882 , piece_id=YDQG4PSVDFQKM2APE2FOMHXDDGWN2UMLS6W3U7BZJFL4CONXTF2A, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.945689447 (501h49m56.816772483s) [noise=true , address=77.90.31.15:4731 , piece_id=HEMHNVQKVNTUDI3YUYUTBFB4KTDU52M2CY5CS6HLDNISY4YO4Q2A, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.946210009 (501h49m56.817293054s) [noise=true , address=77.90.31.18:4815 , piece_id=XMUCIFIPF7J5YT6NNIOTJR5PYSRAWEQX3OFUSKJBQAVBRGRSZBTA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.947998073 (501h49m56.819081108s) [noise=true , address=145.79.227.12:4793 , piece_id=TQQWTNCS7R6Y5NUINKPGJJP6BTLW5S5UPHKLKACCXOHLJFFGYLIQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.949648526 (501h49m56.820731561s) [noise=true , address=77.90.31.13:4625 , piece_id=4SSOYJCUDEWF7DE2KQ5NVHTBUFS55LWCS5HDVMPMQHTK5GOOPJLQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:58.259424303 (501h49m58.130507338s) [upload=440414]: Next returned nil segment Dec 26 13:18:58.259437077 (501h49m58.130520112s) [upload=440414]: Waiting for error group managing segments... Dec 26 13:18:58.593782731 (501h49m58.464865776s) [noise=true , address=193.149.164.143:4702 , piece_id=WCDUGPEQ4OKX5LF6UAKAUI7IIADUGICUWXCKJJZLBQPLR4IQWCMQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:58.821951145 (501h49m58.69303418s) [noise=true , address=145.79.227.8:4356 , piece_id=T4YWHFUJZNM5GCZBF4BF36GUJTZVSNMW3SMBCILKWB4VCQATHDRQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:58.823136852 (501h49m58.694219898s) [noise=true , address=145.79.227.12:4793 , piece_id=TQQWTNCS7R6Y5NUINKPGJJP6BTLW5S5UPHKLKACCXOHLJFFGYLIQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:58.859066810 (501h49m58.730149845s) [noise=true , address=145.79.227.52:4449 , piece_id=SUWWUSJ4NDCMJNRGZ3NU2WMBLISLHYPXJNFB6ISVTH6LHGK6M7OQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:58.908900882 (501h49m58.779983927s) [noise=true , address=145.79.227.40:4054 , piece_id=IZFFBTPOADF6BYIYODQVAVWPUWI2KLKTCRPALMPHGU3ZGLUBXCHA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.096350209 (501h49m58.967433244s) [noise=true , address=77.90.31.18:4815 , piece_id=XMUCIFIPF7J5YT6NNIOTJR5PYSRAWEQX3OFUSKJBQAVBRGRSZBTA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.152026147 (501h49m59.023109182s) [noise=true , address=45.91.116.13:4167 , piece_id=MADAUEX7VYZP7B2I4QCLGGEELNXKWJIC7IN4MO5DOQUQHT7QCKMQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.191341109 (501h49m59.062424144s) [noise=true , address=77.90.31.10:4552 , piece_id=W4HPIARTL67DK24ONOCXV43GDWO5N72VOT3NDIIK222SOQSEOLWA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.363089609 (501h49m59.234172654s) [noise=true , address=77.90.31.15:4731 , piece_id=HEMHNVQKVNTUDI3YUYUTBFB4KTDU52M2CY5CS6HLDNISY4YO4Q2A, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.486136873 (501h49m59.357219908s) [noise=true , address=45.91.116.9:4267 , piece_id=JCL5I57XK4XD6VYK4USM6HA7VI4HMRQ3455PYCJUGS267H5UD5KA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.659904808 (501h49m59.530987854s) [noise=true , address=64.49.13.26:4945 , piece_id=AIHRNVIXZXAHLHRKZ4HC5TX5FAUN33C4UWEXID2LU2RNGIQQSOAA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.746264452 (501h49m59.617347497s) [noise=true , address=145.79.227.44:4218 , piece_id=XMGFRL4GQ5IJX4J4BHTA6EOWUJHQ2BJF47KBBKUSKPRFUFG6GVIQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.762114864 (501h49m59.633197899s) [noise=true , address=145.79.227.33:4897 , piece_id=YGEFUI4Z26M7DQVOGKB2IPCWCQATVNZAAQ6SDGOUA72XA5BOC2JA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.788500227 (501h49m59.659583272s) [noise=true , address=45.91.116.5:4327 , piece_id=K37R3G6TOSNYRHQVD7TCRNXFKLFZ542EWYOLFXHSMEL7UJWZ2GZQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.788524653 (501h49m59.659607698s) [seg_pos={0 0}, upload=440414]: Segment reached optimal threshold of 14 pieces. Dec 26 13:18:59.788737228 (501h49m59.659820273s) [noise=true , address=145.79.227.30:4767 , piece_id=QDGOXWPYO5UCENH6DXG46ONRVDCZORDIV7TDC3FIKYXR4S64Q5MQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 26 13:18:59.788745484 (501h49m59.659828539s) [noise=true , address=145.79.227.30:4767 , piece_id=QDGOXWPYO5UCENH6DXG46ONRVDCZORDIV7TDC3FIKYXR4S64Q5MQ, seg_pos={0 0}, upload=440414]: Long tail cancellation due to optimal threshold Dec 26 13:18:59.788787113 (501h49m59.659870158s) [noise=true , address=77.90.31.13:4625 , piece_id=4SSOYJCUDEWF7DE2KQ5NVHTBUFS55LWCS5HDVMPMQHTK5GOOPJLQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: expected piece hash; context canceled; EOF Dec 26 13:18:59.788810317 (501h49m59.659893352s) [noise=true , address=77.90.31.13:4625 , piece_id=4SSOYJCUDEWF7DE2KQ5NVHTBUFS55LWCS5HDVMPMQHTK5GOOPJLQ, seg_pos={0 0}, upload=440414]: Long tail cancellation due to optimal threshold Dec 26 13:18:59.788981984 (501h49m59.660065029s) [noise=true , address=77.90.31.11:4586 , piece_id=M6IBB3JP5LZP5PMFTSSXHMXBLLJUHLRB5T7UTHE5YWKMR2RNNHPA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2): protocol: expected piece hash; context canceled; EOF Dec 26 13:18:59.788989880 (501h49m59.660072915s) [noise=true , address=77.90.31.11:4586 , piece_id=M6IBB3JP5LZP5PMFTSSXHMXBLLJUHLRB5T7UTHE5YWKMR2RNNHPA, seg_pos={0 0}, upload=440414]: Long tail cancellation due to optimal threshold Dec 26 13:18:59.789136468 (501h49m59.660219503s) [noise=true , address=77.90.31.21:4882 , piece_id=YDQG4PSVDFQKM2APE2FOMHXDDGWN2UMLS6W3U7BZJFL4CONXTF2A, seg_pos={0 0}, upload=440414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: read tcp 188.166.225.170:35658->77.90.31.21:4882: use of closed network connection; conn closed: op Dec 26 13:18:59.789161326 (501h49m59.660244361s) [noise=true , address=77.90.31.21:4882 , piece_id=YDQG4PSVDFQKM2APE2FOMHXDDGWN2UMLS6W3U7BZJFL4CONXTF2A, seg_pos={0 0}, upload=440414]: Long tail cancellation due to optimal threshold Dec 26 13:18:59.789178899 (501h49m59.660261934s) [seg_pos={0 0}, upload=440414]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:18:59.789270323 (501h49m59.660353359s) [upload=440414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:18:59.789275944 (501h49m59.660358989s) [upload=440414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:19:00.090282183 (501h49m59.961365238s) [upload=440414]: Done uploading segments... Dec 26 13:19:00.090312260 (501h49m59.961395305s) [upload=440414]: Upload finished. err: <nil> Duration: 54.369115024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.] duration: 5.789615804s trace: https://jaeger.dev.storj.io/trace/6fe80b7916beddc1 FAIL (2025-12-26 13:20:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 13:20:07.801381656 (501h51m7.672464701s) [upload=440433]: Starting upload Dec 26 13:20:07.801465446 (501h51m7.672548491s) [upload=440433]: Done starting upload Dec 26 13:20:07.801628305 (501h51m7.67271134s) [upload=440433]: Uploading segments... Dec 26 13:20:07.801731422 (501h51m7.672814467s) [upload=440433]: Got next segment. Inline: false Dec 26 13:20:07.801766368 (501h51m7.672849413s) [upload=440433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:20:07.801772310 (501h51m7.672855355s) [upload=440433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:20:08.072384224 (501h51m7.94346727s) [seg_pos={0 0}, upload=440433]: Begin upload segment... Dec 26 13:20:08.072441363 (501h51m7.943524408s) [seg_pos={0 0}, upload=440433]: Done begin upload segment. Dec 26 13:20:08.072476930 (501h51m7.943559975s) [noise=true , address=45.91.116.14:4189 , piece_id=3KXMUYKIRE4B7WLODWAIVEQX6KWNJ6BCEPLTWKBAXCWDNOPMIDXQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.073367006 (501h51m7.944450061s) [noise=true , address=145.79.227.15:4874 , piece_id=Z5NUDX35AQYHU52TSXNOXIZLRE7FU2DFAFHQTLW63A6AQGSXAG7A, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.073856969 (501h51m7.944940014s) [noise=true , address=145.79.227.52:4446 , piece_id=O2DDCC3X4CYNSYPEYPYGEHGYINFTSXNR5D3H7ZVB7B7KWRN6XYSQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.074383511 (501h51m7.945466556s) [noise=true , address=145.79.227.17:4970 , piece_id=KUSLF3FHPA7PSEOQPQ3NXG642AQT2XIVXOXCTY2BHLSDUC4R2FYA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.074809402 (501h51m7.945892447s) [noise=true , address=45.91.116.10:4084 , piece_id=L7XSQR6HKNAXAT7TVMQMA4ZNUYYMIFNZZ6DUV4YEI63NMJDZ67CA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.075207260 (501h51m7.946290305s) [noise=true , address=145.79.227.45:4230 , piece_id=UT535OYV7Q3RKC6XXONQHZCCUPNMZQ3F6KITLTJHUCZUOUCRPL3Q, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.075612612 (501h51m7.946695657s) [noise=true , address=145.79.227.36:4518 , piece_id=MIZW3DDJ5Y7WRJ3GINJG5OHONEEGHZGPMUX2L5S447ELLQ5KOUIA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.076005841 (501h51m7.947088886s) [noise=true , address=77.90.31.10:4519 , piece_id=LRKBH4WZ7QO4DRKEJ45VFFPE2AEXP2GOKGRRXCQQVYL2VHZM3Y5A, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.076417174 (501h51m7.947500229s) [noise=true , address=145.79.227.11:4729 , piece_id=AH3MXLCZTZG6AFKB3XJAJYLJJCAQ3Q52LN2EJTGTGXEN3TBE4EKQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.076854045 (501h51m7.94793709s) [noise=true , address=145.79.227.42:4123 , piece_id=PM2VCKPEINKPJ5FYOYN6N6VOR7ZVUGEBB7YU676KC5QA6VYAL2AQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.077296288 (501h51m7.948379333s) [noise=true , address=145.79.227.20:4438 , piece_id=ML3X6TY2YEVTHGT2FEBPKUHAUHNNXTYMBMSBVPIZR5KHYMN43NVQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.077708924 (501h51m7.948791969s) [noise=true , address=145.79.227.28:4727 , piece_id=KZJEBZXOQXLMJJX2IQHOW2VDTKABVG4IMJMS7UOLVVQFSLMYW4PQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.078220227 (501h51m7.949303272s) [noise=true , address=45.91.116.5:4298 , piece_id=B4O5UNBYINO7G5RY2CS5TVDQLSGZSSQJZF4PWXA5YXI3AG4ASDSA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.078685724 (501h51m7.949768769s) [noise=true , address=45.91.116.6:4973 , piece_id=O5W4B473R5S5SLCVJCDXAKJ7D5W7HLUW4LM3EOEWUZ3RG2DTGODA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.079213168 (501h51m7.950296223s) [noise=true , address=45.91.116.13:4154 , piece_id=56QT63KNE4VPDYXVNB46E72XD3HQDWIJXNEQJUITT4LP3ZGVDALA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.079783023 (501h51m7.950866058s) [noise=true , address=77.90.31.18:4808 , piece_id=ZKBUSBCM3Z4HW566ECL5K7LLHEER2VPFQHAUU3SZ6YZ4VXADSE3Q, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:09.742244476 (501h51m9.613327521s) [upload=440433]: Next returned nil segment Dec 26 13:20:09.742254124 (501h51m9.613337169s) [upload=440433]: Waiting for error group managing segments... Dec 26 13:20:10.086116329 (501h51m9.957199374s) [noise=true , address=145.79.227.52:4446 , piece_id=O2DDCC3X4CYNSYPEYPYGEHGYINFTSXNR5D3H7ZVB7B7KWRN6XYSQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.096844637 (501h51m9.967927682s) [noise=true , address=145.79.227.11:4729 , piece_id=AH3MXLCZTZG6AFKB3XJAJYLJJCAQ3Q52LN2EJTGTGXEN3TBE4EKQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.099963517 (501h51m9.971046562s) [noise=true , address=145.79.227.20:4438 , piece_id=ML3X6TY2YEVTHGT2FEBPKUHAUHNNXTYMBMSBVPIZR5KHYMN43NVQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.117872819 (501h51m9.988955854s) [noise=true , address=145.79.227.42:4123 , piece_id=PM2VCKPEINKPJ5FYOYN6N6VOR7ZVUGEBB7YU676KC5QA6VYAL2AQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.118129789 (501h51m9.989212834s) [noise=true , address=145.79.227.17:4970 , piece_id=KUSLF3FHPA7PSEOQPQ3NXG642AQT2XIVXOXCTY2BHLSDUC4R2FYA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.124768761 (501h51m9.995851806s) [noise=true , address=145.79.227.15:4874 , piece_id=Z5NUDX35AQYHU52TSXNOXIZLRE7FU2DFAFHQTLW63A6AQGSXAG7A, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.134506562 (501h51m10.005589607s) [noise=true , address=145.79.227.45:4230 , piece_id=UT535OYV7Q3RKC6XXONQHZCCUPNMZQ3F6KITLTJHUCZUOUCRPL3Q, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.142973666 (501h51m10.014056711s) [noise=true , address=145.79.227.36:4518 , piece_id=MIZW3DDJ5Y7WRJ3GINJG5OHONEEGHZGPMUX2L5S447ELLQ5KOUIA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.154304121 (501h51m10.025387166s) [noise=true , address=145.79.227.28:4727 , piece_id=KZJEBZXOQXLMJJX2IQHOW2VDTKABVG4IMJMS7UOLVVQFSLMYW4PQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.253347196 (501h51m10.124430242s) [noise=true , address=77.90.31.10:4519 , piece_id=LRKBH4WZ7QO4DRKEJ45VFFPE2AEXP2GOKGRRXCQQVYL2VHZM3Y5A, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.331204631 (501h51m10.202287676s) [noise=true , address=45.91.116.5:4298 , piece_id=B4O5UNBYINO7G5RY2CS5TVDQLSGZSSQJZF4PWXA5YXI3AG4ASDSA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.351837880 (501h51m10.222920925s) [noise=true , address=45.91.116.6:4973 , piece_id=O5W4B473R5S5SLCVJCDXAKJ7D5W7HLUW4LM3EOEWUZ3RG2DTGODA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.370982525 (501h51m10.24206557s) [noise=true , address=45.91.116.10:4084 , piece_id=L7XSQR6HKNAXAT7TVMQMA4ZNUYYMIFNZZ6DUV4YEI63NMJDZ67CA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.371013975 (501h51m10.24209702s) [seg_pos={0 0}, upload=440433]: Segment reached optimal threshold of 13 pieces. Dec 26 13:20:10.371307073 (501h51m10.242390108s) [noise=true , address=45.91.116.13:4154 , piece_id=56QT63KNE4VPDYXVNB46E72XD3HQDWIJXNEQJUITT4LP3ZGVDALA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 13:20:10.371315198 (501h51m10.242398243s) [noise=true , address=45.91.116.13:4154 , piece_id=56QT63KNE4VPDYXVNB46E72XD3HQDWIJXNEQJUITT4LP3ZGVDALA, seg_pos={0 0}, upload=440433]: Long tail cancellation due to optimal threshold Dec 26 13:20:10.371779864 (501h51m10.242862899s) [noise=true , address=45.91.116.14:4189 , piece_id=3KXMUYKIRE4B7WLODWAIVEQX6KWNJ6BCEPLTWKBAXCWDNOPMIDXQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 13:20:10.371787929 (501h51m10.242870974s) [noise=true , address=45.91.116.14:4189 , piece_id=3KXMUYKIRE4B7WLODWAIVEQX6KWNJ6BCEPLTWKBAXCWDNOPMIDXQ, seg_pos={0 0}, upload=440433]: Long tail cancellation due to optimal threshold Dec 26 13:20:10.371911144 (501h51m10.242994189s) [noise=true , address=77.90.31.18:4808 , piece_id=ZKBUSBCM3Z4HW566ECL5K7LLHEER2VPFQHAUU3SZ6YZ4VXADSE3Q, seg_pos={0 0}, upload=440433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:37386->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:37386->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37386->77.90.31.18:4808: use of closed network connection Dec 26 13:20:10.371918107 (501h51m10.243001152s) [noise=true , address=77.90.31.18:4808 , piece_id=ZKBUSBCM3Z4HW566ECL5K7LLHEER2VPFQHAUU3SZ6YZ4VXADSE3Q, seg_pos={0 0}, upload=440433]: Long tail cancellation due to optimal threshold Dec 26 13:20:10.371944227 (501h51m10.243027262s) [seg_pos={0 0}, upload=440433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:20:10.372035760 (501h51m10.243118796s) [upload=440433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:20:10.372042092 (501h51m10.243125137s) [upload=440433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:20:10.665509598 (501h51m10.536592653s) [upload=440433]: Done uploading segments... Dec 26 13:20:10.665545947 (501h51m10.536628992s) [upload=440433]: Upload finished. err: <nil> Duration: 5.789634229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X......XX] duration: 30.002834099s trace: https://jaeger.dev.storj.io/trace/6fe80b7916beddc1 FAIL (2025-12-26 13:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 13:20:07.801381656 (501h51m7.672464701s) [upload=440433]: Starting upload Dec 26 13:20:07.801465446 (501h51m7.672548491s) [upload=440433]: Done starting upload Dec 26 13:20:07.801628305 (501h51m7.67271134s) [upload=440433]: Uploading segments... Dec 26 13:20:07.801731422 (501h51m7.672814467s) [upload=440433]: Got next segment. Inline: false Dec 26 13:20:07.801766368 (501h51m7.672849413s) [upload=440433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:20:07.801772310 (501h51m7.672855355s) [upload=440433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:20:08.072384224 (501h51m7.94346727s) [seg_pos={0 0}, upload=440433]: Begin upload segment... Dec 26 13:20:08.072441363 (501h51m7.943524408s) [seg_pos={0 0}, upload=440433]: Done begin upload segment. Dec 26 13:20:08.072476930 (501h51m7.943559975s) [noise=true , address=45.91.116.14:4189 , piece_id=3KXMUYKIRE4B7WLODWAIVEQX6KWNJ6BCEPLTWKBAXCWDNOPMIDXQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.073367006 (501h51m7.944450061s) [noise=true , address=145.79.227.15:4874 , piece_id=Z5NUDX35AQYHU52TSXNOXIZLRE7FU2DFAFHQTLW63A6AQGSXAG7A, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.073856969 (501h51m7.944940014s) [noise=true , address=145.79.227.52:4446 , piece_id=O2DDCC3X4CYNSYPEYPYGEHGYINFTSXNR5D3H7ZVB7B7KWRN6XYSQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.074383511 (501h51m7.945466556s) [noise=true , address=145.79.227.17:4970 , piece_id=KUSLF3FHPA7PSEOQPQ3NXG642AQT2XIVXOXCTY2BHLSDUC4R2FYA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.074809402 (501h51m7.945892447s) [noise=true , address=45.91.116.10:4084 , piece_id=L7XSQR6HKNAXAT7TVMQMA4ZNUYYMIFNZZ6DUV4YEI63NMJDZ67CA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.075207260 (501h51m7.946290305s) [noise=true , address=145.79.227.45:4230 , piece_id=UT535OYV7Q3RKC6XXONQHZCCUPNMZQ3F6KITLTJHUCZUOUCRPL3Q, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.075612612 (501h51m7.946695657s) [noise=true , address=145.79.227.36:4518 , piece_id=MIZW3DDJ5Y7WRJ3GINJG5OHONEEGHZGPMUX2L5S447ELLQ5KOUIA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.076005841 (501h51m7.947088886s) [noise=true , address=77.90.31.10:4519 , piece_id=LRKBH4WZ7QO4DRKEJ45VFFPE2AEXP2GOKGRRXCQQVYL2VHZM3Y5A, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.076417174 (501h51m7.947500229s) [noise=true , address=145.79.227.11:4729 , piece_id=AH3MXLCZTZG6AFKB3XJAJYLJJCAQ3Q52LN2EJTGTGXEN3TBE4EKQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.076854045 (501h51m7.94793709s) [noise=true , address=145.79.227.42:4123 , piece_id=PM2VCKPEINKPJ5FYOYN6N6VOR7ZVUGEBB7YU676KC5QA6VYAL2AQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.077296288 (501h51m7.948379333s) [noise=true , address=145.79.227.20:4438 , piece_id=ML3X6TY2YEVTHGT2FEBPKUHAUHNNXTYMBMSBVPIZR5KHYMN43NVQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.077708924 (501h51m7.948791969s) [noise=true , address=145.79.227.28:4727 , piece_id=KZJEBZXOQXLMJJX2IQHOW2VDTKABVG4IMJMS7UOLVVQFSLMYW4PQ, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.078220227 (501h51m7.949303272s) [noise=true , address=45.91.116.5:4298 , piece_id=B4O5UNBYINO7G5RY2CS5TVDQLSGZSSQJZF4PWXA5YXI3AG4ASDSA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.078685724 (501h51m7.949768769s) [noise=true , address=45.91.116.6:4973 , piece_id=O5W4B473R5S5SLCVJCDXAKJ7D5W7HLUW4LM3EOEWUZ3RG2DTGODA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.079213168 (501h51m7.950296223s) [noise=true , address=45.91.116.13:4154 , piece_id=56QT63KNE4VPDYXVNB46E72XD3HQDWIJXNEQJUITT4LP3ZGVDALA, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:08.079783023 (501h51m7.950866058s) [noise=true , address=77.90.31.18:4808 , piece_id=ZKBUSBCM3Z4HW566ECL5K7LLHEER2VPFQHAUU3SZ6YZ4VXADSE3Q, seg_pos={0 0}, upload=440433]: Uploading piece... Dec 26 13:20:09.742244476 (501h51m9.613327521s) [upload=440433]: Next returned nil segment Dec 26 13:20:09.742254124 (501h51m9.613337169s) [upload=440433]: Waiting for error group managing segments... Dec 26 13:20:10.086116329 (501h51m9.957199374s) [noise=true , address=145.79.227.52:4446 , piece_id=O2DDCC3X4CYNSYPEYPYGEHGYINFTSXNR5D3H7ZVB7B7KWRN6XYSQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.096844637 (501h51m9.967927682s) [noise=true , address=145.79.227.11:4729 , piece_id=AH3MXLCZTZG6AFKB3XJAJYLJJCAQ3Q52LN2EJTGTGXEN3TBE4EKQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.099963517 (501h51m9.971046562s) [noise=true , address=145.79.227.20:4438 , piece_id=ML3X6TY2YEVTHGT2FEBPKUHAUHNNXTYMBMSBVPIZR5KHYMN43NVQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.117872819 (501h51m9.988955854s) [noise=true , address=145.79.227.42:4123 , piece_id=PM2VCKPEINKPJ5FYOYN6N6VOR7ZVUGEBB7YU676KC5QA6VYAL2AQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.118129789 (501h51m9.989212834s) [noise=true , address=145.79.227.17:4970 , piece_id=KUSLF3FHPA7PSEOQPQ3NXG642AQT2XIVXOXCTY2BHLSDUC4R2FYA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.124768761 (501h51m9.995851806s) [noise=true , address=145.79.227.15:4874 , piece_id=Z5NUDX35AQYHU52TSXNOXIZLRE7FU2DFAFHQTLW63A6AQGSXAG7A, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.134506562 (501h51m10.005589607s) [noise=true , address=145.79.227.45:4230 , piece_id=UT535OYV7Q3RKC6XXONQHZCCUPNMZQ3F6KITLTJHUCZUOUCRPL3Q, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.142973666 (501h51m10.014056711s) [noise=true , address=145.79.227.36:4518 , piece_id=MIZW3DDJ5Y7WRJ3GINJG5OHONEEGHZGPMUX2L5S447ELLQ5KOUIA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.154304121 (501h51m10.025387166s) [noise=true , address=145.79.227.28:4727 , piece_id=KZJEBZXOQXLMJJX2IQHOW2VDTKABVG4IMJMS7UOLVVQFSLMYW4PQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.253347196 (501h51m10.124430242s) [noise=true , address=77.90.31.10:4519 , piece_id=LRKBH4WZ7QO4DRKEJ45VFFPE2AEXP2GOKGRRXCQQVYL2VHZM3Y5A, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.331204631 (501h51m10.202287676s) [noise=true , address=45.91.116.5:4298 , piece_id=B4O5UNBYINO7G5RY2CS5TVDQLSGZSSQJZF4PWXA5YXI3AG4ASDSA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.351837880 (501h51m10.222920925s) [noise=true , address=45.91.116.6:4973 , piece_id=O5W4B473R5S5SLCVJCDXAKJ7D5W7HLUW4LM3EOEWUZ3RG2DTGODA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.370982525 (501h51m10.24206557s) [noise=true , address=45.91.116.10:4084 , piece_id=L7XSQR6HKNAXAT7TVMQMA4ZNUYYMIFNZZ6DUV4YEI63NMJDZ67CA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: <nil> Dec 26 13:20:10.371013975 (501h51m10.24209702s) [seg_pos={0 0}, upload=440433]: Segment reached optimal threshold of 13 pieces. Dec 26 13:20:10.371307073 (501h51m10.242390108s) [noise=true , address=45.91.116.13:4154 , piece_id=56QT63KNE4VPDYXVNB46E72XD3HQDWIJXNEQJUITT4LP3ZGVDALA, seg_pos={0 0}, upload=440433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 13:20:10.371315198 (501h51m10.242398243s) [noise=true , address=45.91.116.13:4154 , piece_id=56QT63KNE4VPDYXVNB46E72XD3HQDWIJXNEQJUITT4LP3ZGVDALA, seg_pos={0 0}, upload=440433]: Long tail cancellation due to optimal threshold Dec 26 13:20:10.371779864 (501h51m10.242862899s) [noise=true , address=45.91.116.14:4189 , piece_id=3KXMUYKIRE4B7WLODWAIVEQX6KWNJ6BCEPLTWKBAXCWDNOPMIDXQ, seg_pos={0 0}, upload=440433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 13:20:10.371787929 (501h51m10.242870974s) [noise=true , address=45.91.116.14:4189 , piece_id=3KXMUYKIRE4B7WLODWAIVEQX6KWNJ6BCEPLTWKBAXCWDNOPMIDXQ, seg_pos={0 0}, upload=440433]: Long tail cancellation due to optimal threshold Dec 26 13:20:10.371911144 (501h51m10.242994189s) [noise=true , address=77.90.31.18:4808 , piece_id=ZKBUSBCM3Z4HW566ECL5K7LLHEER2VPFQHAUU3SZ6YZ4VXADSE3Q, seg_pos={0 0}, upload=440433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:37386->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:37386->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37386->77.90.31.18:4808: use of closed network connection Dec 26 13:20:10.371918107 (501h51m10.243001152s) [noise=true , address=77.90.31.18:4808 , piece_id=ZKBUSBCM3Z4HW566ECL5K7LLHEER2VPFQHAUU3SZ6YZ4VXADSE3Q, seg_pos={0 0}, upload=440433]: Long tail cancellation due to optimal threshold Dec 26 13:20:10.371944227 (501h51m10.243027262s) [seg_pos={0 0}, upload=440433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:20:10.372035760 (501h51m10.243118796s) [upload=440433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:20:10.372042092 (501h51m10.243125137s) [upload=440433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:20:10.665509598 (501h51m10.536592653s) [upload=440433]: Done uploading segments... Dec 26 13:20:10.665545947 (501h51m10.536628992s) [upload=440433]: Upload finished. err: <nil> Duration: 30.002869215s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.] duration: 3.387701781s trace: https://jaeger.dev.storj.io/trace/275d7900dd51c85a FAIL (2025-12-26 13:20:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.387758378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [................X.XXX...X..X...................XX.X........X] duration: 1m0.003080036s trace: https://jaeger.dev.storj.io/trace/3764360f5fc66fe4 FAIL (2025-12-26 13:19:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo, Piece ID: HEMHNVQKVNTUDI3YUYUTBFB4KTDU52M2CY5CS6HLDNISY4YO4Q2A) context deadline exceeded; piecestore: (Node ID: 1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, Piece ID: JCL5I57XK4XD6VYK4USM6HA7VI4HMRQ3455PYCJUGS267H5UD5KA) context deadline exceeded; piecestore: (Node ID: 12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, Piece ID: K37R3G6TOSNYRHQVD7TCRNXFKLFZ542EWYOLFXHSMEL7UJWZ2GZQ) context deadline exceeded ======LOGS====== .91.116.9:4267 , piece_id=JCL5I57XK4XD6VYK4USM6HA7VI4HMRQ3455PYCJUGS267H5UD5KA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.932725752 (501h49m56.803808797s) [noise=true , address=145.79.227.40:4054 , piece_id=IZFFBTPOADF6BYIYODQVAVWPUWI2KLKTCRPALMPHGU3ZGLUBXCHA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.933157144 (501h49m56.804240179s) [noise=true , address=45.91.116.5:4327 , piece_id=K37R3G6TOSNYRHQVD7TCRNXFKLFZ542EWYOLFXHSMEL7UJWZ2GZQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.933517910 (501h49m56.804600945s) [noise=true , address=45.91.116.13:4167 , piece_id=MADAUEX7VYZP7B2I4QCLGGEELNXKWJIC7IN4MO5DOQUQHT7QCKMQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.933929144 (501h49m56.805012179s) [noise=true , address=77.90.31.10:4552 , piece_id=W4HPIARTL67DK24ONOCXV43GDWO5N72VOT3NDIIK222SOQSEOLWA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.934322603 (501h49m56.805405648s) [noise=true , address=145.79.227.8:4356 , piece_id=T4YWHFUJZNM5GCZBF4BF36GUJTZVSNMW3SMBCILKWB4VCQATHDRQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.934734637 (501h49m56.805817682s) [noise=true , address=193.149.164.143:4702 , piece_id=WCDUGPEQ4OKX5LF6UAKAUI7IIADUGICUWXCKJJZLBQPLR4IQWCMQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.935094532 (501h49m56.806177577s) [noise=true , address=145.79.227.33:4897 , piece_id=YGEFUI4Z26M7DQVOGKB2IPCWCQATVNZAAQ6SDGOUA72XA5BOC2JA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.936684089 (501h49m56.807767124s) [noise=true , address=145.79.227.44:4218 , piece_id=XMGFRL4GQ5IJX4J4BHTA6EOWUJHQ2BJF47KBBKUSKPRFUFG6GVIQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.937029768 (501h49m56.808112813s) [noise=true , address=145.79.227.30:4767 , piece_id=QDGOXWPYO5UCENH6DXG46ONRVDCZORDIV7TDC3FIKYXR4S64Q5MQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.938672836 (501h49m56.809755871s) [noise=true , address=64.49.13.26:4945 , piece_id=AIHRNVIXZXAHLHRKZ4HC5TX5FAUN33C4UWEXID2LU2RNGIQQSOAA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.942723659 (501h49m56.813806704s) [noise=true , address=145.79.227.52:4449 , piece_id=SUWWUSJ4NDCMJNRGZ3NU2WMBLISLHYPXJNFB6ISVTH6LHGK6M7OQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.944292236 (501h49m56.815375271s) [noise=true , address=77.90.31.21:4882 , piece_id=YDQG4PSVDFQKM2APE2FOMHXDDGWN2UMLS6W3U7BZJFL4CONXTF2A, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.945689447 (501h49m56.816772483s) [noise=true , address=77.90.31.15:4731 , piece_id=HEMHNVQKVNTUDI3YUYUTBFB4KTDU52M2CY5CS6HLDNISY4YO4Q2A, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.946210009 (501h49m56.817293054s) [noise=true , address=77.90.31.18:4815 , piece_id=XMUCIFIPF7J5YT6NNIOTJR5PYSRAWEQX3OFUSKJBQAVBRGRSZBTA, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.947998073 (501h49m56.819081108s) [noise=true , address=145.79.227.12:4793 , piece_id=TQQWTNCS7R6Y5NUINKPGJJP6BTLW5S5UPHKLKACCXOHLJFFGYLIQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:56.949648526 (501h49m56.820731561s) [noise=true , address=77.90.31.13:4625 , piece_id=4SSOYJCUDEWF7DE2KQ5NVHTBUFS55LWCS5HDVMPMQHTK5GOOPJLQ, seg_pos={0 0}, upload=440414]: Uploading piece... Dec 26 13:18:58.259424303 (501h49m58.130507338s) [upload=440414]: Next returned nil segment Dec 26 13:18:58.259437077 (501h49m58.130520112s) [upload=440414]: Waiting for error group managing segments... Dec 26 13:18:58.593782731 (501h49m58.464865776s) [noise=true , address=193.149.164.143:4702 , piece_id=WCDUGPEQ4OKX5LF6UAKAUI7IIADUGICUWXCKJJZLBQPLR4IQWCMQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:58.821951145 (501h49m58.69303418s) [noise=true , address=145.79.227.8:4356 , piece_id=T4YWHFUJZNM5GCZBF4BF36GUJTZVSNMW3SMBCILKWB4VCQATHDRQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:58.823136852 (501h49m58.694219898s) [noise=true , address=145.79.227.12:4793 , piece_id=TQQWTNCS7R6Y5NUINKPGJJP6BTLW5S5UPHKLKACCXOHLJFFGYLIQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:58.859066810 (501h49m58.730149845s) [noise=true , address=145.79.227.52:4449 , piece_id=SUWWUSJ4NDCMJNRGZ3NU2WMBLISLHYPXJNFB6ISVTH6LHGK6M7OQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:58.908900882 (501h49m58.779983927s) [noise=true , address=145.79.227.40:4054 , piece_id=IZFFBTPOADF6BYIYODQVAVWPUWI2KLKTCRPALMPHGU3ZGLUBXCHA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.096350209 (501h49m58.967433244s) [noise=true , address=77.90.31.18:4815 , piece_id=XMUCIFIPF7J5YT6NNIOTJR5PYSRAWEQX3OFUSKJBQAVBRGRSZBTA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.152026147 (501h49m59.023109182s) [noise=true , address=45.91.116.13:4167 , piece_id=MADAUEX7VYZP7B2I4QCLGGEELNXKWJIC7IN4MO5DOQUQHT7QCKMQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.191341109 (501h49m59.062424144s) [noise=true , address=77.90.31.10:4552 , piece_id=W4HPIARTL67DK24ONOCXV43GDWO5N72VOT3NDIIK222SOQSEOLWA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.363089609 (501h49m59.234172654s) [noise=true , address=77.90.31.15:4731 , piece_id=HEMHNVQKVNTUDI3YUYUTBFB4KTDU52M2CY5CS6HLDNISY4YO4Q2A, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.486136873 (501h49m59.357219908s) [noise=true , address=45.91.116.9:4267 , piece_id=JCL5I57XK4XD6VYK4USM6HA7VI4HMRQ3455PYCJUGS267H5UD5KA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.659904808 (501h49m59.530987854s) [noise=true , address=64.49.13.26:4945 , piece_id=AIHRNVIXZXAHLHRKZ4HC5TX5FAUN33C4UWEXID2LU2RNGIQQSOAA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.746264452 (501h49m59.617347497s) [noise=true , address=145.79.227.44:4218 , piece_id=XMGFRL4GQ5IJX4J4BHTA6EOWUJHQ2BJF47KBBKUSKPRFUFG6GVIQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.762114864 (501h49m59.633197899s) [noise=true , address=145.79.227.33:4897 , piece_id=YGEFUI4Z26M7DQVOGKB2IPCWCQATVNZAAQ6SDGOUA72XA5BOC2JA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.788500227 (501h49m59.659583272s) [noise=true , address=45.91.116.5:4327 , piece_id=K37R3G6TOSNYRHQVD7TCRNXFKLFZ542EWYOLFXHSMEL7UJWZ2GZQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: <nil> Dec 26 13:18:59.788524653 (501h49m59.659607698s) [seg_pos={0 0}, upload=440414]: Segment reached optimal threshold of 14 pieces. Dec 26 13:18:59.788737228 (501h49m59.659820273s) [noise=true , address=145.79.227.30:4767 , piece_id=QDGOXWPYO5UCENH6DXG46ONRVDCZORDIV7TDC3FIKYXR4S64Q5MQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 26 13:18:59.788745484 (501h49m59.659828539s) [noise=true , address=145.79.227.30:4767 , piece_id=QDGOXWPYO5UCENH6DXG46ONRVDCZORDIV7TDC3FIKYXR4S64Q5MQ, seg_pos={0 0}, upload=440414]: Long tail cancellation due to optimal threshold Dec 26 13:18:59.788787113 (501h49m59.659870158s) [noise=true , address=77.90.31.13:4625 , piece_id=4SSOYJCUDEWF7DE2KQ5NVHTBUFS55LWCS5HDVMPMQHTK5GOOPJLQ, seg_pos={0 0}, upload=440414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: expected piece hash; context canceled; EOF Dec 26 13:18:59.788810317 (501h49m59.659893352s) [noise=true , address=77.90.31.13:4625 , piece_id=4SSOYJCUDEWF7DE2KQ5NVHTBUFS55LWCS5HDVMPMQHTK5GOOPJLQ, seg_pos={0 0}, upload=440414]: Long tail cancellation due to optimal threshold Dec 26 13:18:59.788981984 (501h49m59.660065029s) [noise=true , address=77.90.31.11:4586 , piece_id=M6IBB3JP5LZP5PMFTSSXHMXBLLJUHLRB5T7UTHE5YWKMR2RNNHPA, seg_pos={0 0}, upload=440414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2): protocol: expected piece hash; context canceled; EOF Dec 26 13:18:59.788989880 (501h49m59.660072915s) [noise=true , address=77.90.31.11:4586 , piece_id=M6IBB3JP5LZP5PMFTSSXHMXBLLJUHLRB5T7UTHE5YWKMR2RNNHPA, seg_pos={0 0}, upload=440414]: Long tail cancellation due to optimal threshold Dec 26 13:18:59.789136468 (501h49m59.660219503s) [noise=true , address=77.90.31.21:4882 , piece_id=YDQG4PSVDFQKM2APE2FOMHXDDGWN2UMLS6W3U7BZJFL4CONXTF2A, seg_pos={0 0}, upload=440414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: read tcp 188.166.225.170:35658->77.90.31.21:4882: use of closed network connection; conn closed: op Dec 26 13:18:59.789161326 (501h49m59.660244361s) [noise=true , address=77.90.31.21:4882 , piece_id=YDQG4PSVDFQKM2APE2FOMHXDDGWN2UMLS6W3U7BZJFL4CONXTF2A, seg_pos={0 0}, upload=440414]: Long tail cancellation due to optimal threshold Dec 26 13:18:59.789178899 (501h49m59.660261934s) [seg_pos={0 0}, upload=440414]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:18:59.789270323 (501h49m59.660353359s) [upload=440414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:18:59.789275944 (501h49m59.660358989s) [upload=440414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:19:00.090282183 (501h49m59.961365238s) [upload=440414]: Done uploading segments... Dec 26 13:19:00.090312260 (501h49m59.961395305s) [upload=440414]: Upload finished. err: <nil> Duration: 1m0.003091638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X......X] duration: 30.004334978s trace: https://jaeger.dev.storj.io/trace/44918f65dfcf63e7 FAIL (2025-12-26 13:19:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 13:19:19.188922944 (501h50m19.060005979s) [upload=440421]: Starting upload Dec 26 13:19:19.188977067 (501h50m19.060060102s) [upload=440421]: Done starting upload Dec 26 13:19:19.189191295 (501h50m19.06027434s) [upload=440421]: Uploading segments... Dec 26 13:19:19.189452883 (501h50m19.060535919s) [upload=440421]: Got next segment. Inline: false Dec 26 13:19:19.189480666 (501h50m19.060563711s) [upload=440421]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:19:19.189484684 (501h50m19.060567719s) [upload=440421]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:19:19.432875293 (501h50m19.303958348s) [seg_pos={0 0}, upload=440421]: Begin upload segment... Dec 26 13:19:19.433043403 (501h50m19.304126448s) [seg_pos={0 0}, upload=440421]: Done begin upload segment. Dec 26 13:19:19.433070325 (501h50m19.30415337s) [noise=true , address=77.90.31.10:4519 , piece_id=Z3SH4X3OOQ54EIFAOI7DNOFH4Q4XUJZNBI6PLHYR5T235MYCNCVA, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.433111683 (501h50m19.304194728s) [noise=true , address=145.79.227.21:4475 , piece_id=COBS6XG7LAGABPPZJ3EB4ZBVH4FGGV3LSQVF4KP5OCB5JS5WEMJQ, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.433759618 (501h50m19.304842673s) [noise=true , address=45.91.116.12:4225 , piece_id=XVY27MXSJMMW7YUCHRLB7UEVKOPCVP5P52JED5WSWV4CFGB3ZNQA, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.434287312 (501h50m19.305370357s) [noise=true , address=145.79.227.49:4337 , piece_id=M34IEPPMGI5D7JWI66GUUKT625RZ5C3FGDJUH5LOREXSZES4HFVA, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.435706805 (501h50m19.30678985s) [noise=true , address=145.79.227.20:4438 , piece_id=WQ2O5BKBK6HTVXREJYQAMPLUOECDEEBVSHGUIQATWKBTEZPGUKAQ, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.436229641 (501h50m19.307312686s) [noise=true , address=145.79.227.46:4601 , piece_id=ICGIYZLCIYWDR4ZA3ZIDGHUEMDAM3CAERVUZU554A2NJAVIM447Q, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.436746314 (501h50m19.307829359s) [noise=true , address=145.79.227.4:4186 , piece_id=QUB7X4P22XCWWUSYDIS3Y556NUXRP7LZ2NIQNM5RLV2DZOSEAWOA, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.438474535 (501h50m19.30955758s) [noise=true , address=145.79.227.11:4729 , piece_id=TNXNVNA5CYIPBDS5N6CJZUFBRJTVQDGRIODOFARMN7MZWM65RNWA, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.438572181 (501h50m19.309655226s) [noise=true , address=145.79.227.29:4941 , piece_id=2KNINH2XES6AFFF2KGAAIU5SJCMC4U2MPGCWNDBLP5T74NJN2ITA, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.440762702 (501h50m19.311845737s) [noise=true , address=145.79.227.15:4874 , piece_id=IKG3ED26XJ5K6DPJT3XDNPL4IOWFQW6B3DLOYVGQNVZDIP7O3DLQ, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.443977183 (501h50m19.315060238s) [noise=true , address=145.79.227.45:4230 , piece_id=UIOKKXPN557F4KMVU7WDVPDW6234Q76HRUJWVD3XRK2OZEY6IA6A, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.445677922 (501h50m19.316760967s) [noise=true , address=145.79.227.12:4765 , piece_id=HIPWKTY2NHRJHHNBTNAH3AUWXBMYNZNBGGTVLOBXJCZNPOMNRQSA, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.445777491 (501h50m19.316860536s) [noise=true , address=145.79.227.2:4113 , piece_id=YMU5Q7RKTCIY5BZMALG2KSMZKV2MBG6WE7VZADXGEHCE4BKZLCUQ, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.447579513 (501h50m19.318662558s) [noise=true , address=45.91.116.13:4155 , piece_id=EGFNVBEE2DOBE7ZU6Y3VZKG4NLQHWLJGPCJCA6YIRDV3UMMKB52A, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.449354834 (501h50m19.320437879s) [noise=true , address=45.91.116.10:4084 , piece_id=ES43YEE7AWOQBZ4RCR2X4HTU7WPYOW4YHKFPGWPUUKW34QUL5DVA, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:19.451155792 (501h50m19.322238837s) [noise=true , address=77.90.31.11:4555 , piece_id=PHOOWAJKAYQKHWQYLRIFOBA4GNGH7UF2BCVQCRH3FCC3NTMKNX7Q, seg_pos={0 0}, upload=440421]: Uploading piece... Dec 26 13:19:21.153768742 (501h50m21.024851807s) [upload=440421]: Next returned nil segment Dec 26 13:19:21.153884412 (501h50m21.024967468s) [upload=440421]: Waiting for error group managing segments... Dec 26 13:19:21.439225993 (501h50m21.310309028s) [noise=true , address=145.79.227.12:4765 , piece_id=HIPWKTY2NHRJHHNBTNAH3AUWXBMYNZNBGGTVLOBXJCZNPOMNRQSA, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.443781578 (501h50m21.314864623s) [noise=true , address=145.79.227.21:4475 , piece_id=COBS6XG7LAGABPPZJ3EB4ZBVH4FGGV3LSQVF4KP5OCB5JS5WEMJQ, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.451805978 (501h50m21.322889013s) [noise=true , address=145.79.227.15:4874 , piece_id=IKG3ED26XJ5K6DPJT3XDNPL4IOWFQW6B3DLOYVGQNVZDIP7O3DLQ, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.452691714 (501h50m21.323774759s) [noise=true , address=145.79.227.45:4230 , piece_id=UIOKKXPN557F4KMVU7WDVPDW6234Q76HRUJWVD3XRK2OZEY6IA6A, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.453880137 (501h50m21.324963182s) [noise=true , address=145.79.227.46:4601 , piece_id=ICGIYZLCIYWDR4ZA3ZIDGHUEMDAM3CAERVUZU554A2NJAVIM447Q, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.455969756 (501h50m21.327052791s) [noise=true , address=145.79.227.11:4729 , piece_id=TNXNVNA5CYIPBDS5N6CJZUFBRJTVQDGRIODOFARMN7MZWM65RNWA, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.458213820 (501h50m21.329296855s) [noise=true , address=145.79.227.4:4186 , piece_id=QUB7X4P22XCWWUSYDIS3Y556NUXRP7LZ2NIQNM5RLV2DZOSEAWOA, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.465136642 (501h50m21.336219677s) [noise=true , address=145.79.227.49:4337 , piece_id=M34IEPPMGI5D7JWI66GUUKT625RZ5C3FGDJUH5LOREXSZES4HFVA, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.482421235 (501h50m21.353504291s) [noise=true , address=145.79.227.29:4941 , piece_id=2KNINH2XES6AFFF2KGAAIU5SJCMC4U2MPGCWNDBLP5T74NJN2ITA, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.485173546 (501h50m21.356256611s) [noise=true , address=145.79.227.20:4438 , piece_id=WQ2O5BKBK6HTVXREJYQAMPLUOECDEEBVSHGUIQATWKBTEZPGUKAQ, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.485539863 (501h50m21.356622898s) [noise=true , address=145.79.227.2:4113 , piece_id=YMU5Q7RKTCIY5BZMALG2KSMZKV2MBG6WE7VZADXGEHCE4BKZLCUQ, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.685246200 (501h50m21.556329245s) [noise=true , address=77.90.31.10:4519 , piece_id=Z3SH4X3OOQ54EIFAOI7DNOFH4Q4XUJZNBI6PLHYR5T235MYCNCVA, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.717095847 (501h50m21.588178892s) [noise=true , address=45.91.116.10:4084 , piece_id=ES43YEE7AWOQBZ4RCR2X4HTU7WPYOW4YHKFPGWPUUKW34QUL5DVA, seg_pos={0 0}, upload=440421]: Done uploading piece. err: <nil> Dec 26 13:19:21.717117739 (501h50m21.588200784s) [seg_pos={0 0}, upload=440421]: Segment reached optimal threshold of 13 pieces. Dec 26 13:19:21.717309645 (501h50m21.58839269s) [noise=true , address=45.91.116.12:4225 , piece_id=XVY27MXSJMMW7YUCHRLB7UEVKOPCVP5P52JED5WSWV4CFGB3ZNQA, seg_pos={0 0}, upload=440421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 13:19:21.717316157 (501h50m21.588399212s) [noise=true , address=45.91.116.12:4225 , piece_id=XVY27MXSJMMW7YUCHRLB7UEVKOPCVP5P52JED5WSWV4CFGB3ZNQA, seg_pos={0 0}, upload=440421]: Long tail cancellation due to optimal threshold Dec 26 13:19:21.717558719 (501h50m21.588641764s) [noise=true , address=45.91.116.13:4155 , piece_id=EGFNVBEE2DOBE7ZU6Y3VZKG4NLQHWLJGPCJCA6YIRDV3UMMKB52A, seg_pos={0 0}, upload=440421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:35782->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:35782->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35782->45.91.116.13:4155: use of closed network connection Dec 26 13:19:21.717565662 (501h50m21.588648697s) [noise=true , address=45.91.116.13:4155 , piece_id=EGFNVBEE2DOBE7ZU6Y3VZKG4NLQHWLJGPCJCA6YIRDV3UMMKB52A, seg_pos={0 0}, upload=440421]: Long tail cancellation due to optimal threshold Dec 26 13:19:21.717683797 (501h50m21.588766842s) [noise=true , address=77.90.31.11:4555 , piece_id=PHOOWAJKAYQKHWQYLRIFOBA4GNGH7UF2BCVQCRH3FCC3NTMKNX7Q, seg_pos={0 0}, upload=440421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 13:19:21.717689729 (501h50m21.588772774s) [noise=true , address=77.90.31.11:4555 , piece_id=PHOOWAJKAYQKHWQYLRIFOBA4GNGH7UF2BCVQCRH3FCC3NTMKNX7Q, seg_pos={0 0}, upload=440421]: Long tail cancellation due to optimal threshold Dec 26 13:19:21.717705999 (501h50m21.588789034s) [seg_pos={0 0}, upload=440421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:19:21.717779169 (501h50m21.588862214s) [upload=440421]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:19:21.717784929 (501h50m21.588867974s) [upload=440421]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:19:22.002221037 (501h50m21.873304072s) [upload=440421]: Done uploading segments... Dec 26 13:19:22.002241605 (501h50m21.87332464s) [upload=440421]: Upload finished. err: <nil> Duration: 30.004358935s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X] duration: 5.00127431s trace: https://jaeger.dev.storj.io/trace/275d7900dd51c85a FAIL (2025-12-26 13:19:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001298416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.] duration: 4.161766598s trace: https://jaeger.dev.storj.io/trace/4125c7bf904725df FAIL (2025-12-26 13:18:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.161791805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.................X.XXX...X..X...................XX.X........] duration: 48.272875962s trace: https://jaeger.dev.storj.io/trace/50ff20dcdf4be9b4 FAIL (2025-12-26 13:18:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (501h42m7.38824738s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517572723 (501h42m7.388655758s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517971292 (501h42m7.389054327s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518432961 (501h42m7.389516006s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518809027 (501h42m7.389892062s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519282127 (501h42m7.390365172s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519624399 (501h42m7.390707444s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519979325 (501h42m7.39106236s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.520359489 (501h42m7.391442534s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.521627654 (501h42m7.392710689s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.523533252 (501h42m7.394616287s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.524846152 (501h42m7.395929188s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.526350467 (501h42m7.397433542s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.528102293 (501h42m7.399185328s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.529588312 (501h42m7.400671347s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:08.839991300 (501h42m8.711074345s) [upload=440304]: Next returned nil segment Dec 26 13:11:08.840016208 (501h42m8.711099253s) [upload=440304]: Waiting for error group managing segments... Dec 26 13:11:09.181763785 (501h42m9.05284683s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.294980239 (501h42m9.166063284s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.435548195 (501h42m9.30663124s) [noise=true , address=145.79.227.28:4745 , piece_id=ICIDFSLXT62MLRKPE7NAJI4VOW6SLVMJL4KTNHHRJJK5XDAHBUWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.436737098 (501h42m9.307820143s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.439665284 (501h42m9.310748329s) [noise=true , address=145.79.227.40:4066 , piece_id=RPRP2QP7CPSCWSUI4DYVJ65TX5W2TLLBAHLFHJ7OSK4M62OXA6WA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.468255686 (501h42m9.339338731s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.758044865 (501h42m9.62912791s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.361420468 (501h42m10.232503513s) [noise=true , address=145.79.227.30:4790 , piece_id=6RSHKA5O3KNIPZG3EZD6KZZG7T4XFHRYYW7GBCSYE7G4JIWPBCNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.865755260 (501h42m10.736838295s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.384330849 (501h42m11.255413884s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.822065002 (501h42m11.693148047s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.921774887 (501h42m11.792857932s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:12.562307844 (501h42m12.433390889s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166097725 (501h42m13.03718077s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166117133 (501h42m13.037200178s) [seg_pos={0 0}, upload=440304]: Segment reached optimal threshold of 14 pieces. Dec 26 13:11:13.166391525 (501h42m13.03747457s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection Dec 26 13:11:13.166398408 (501h42m13.037481443s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166570326 (501h42m13.037653361s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d): protocol: expected piece hash; context canceled; EOF Dec 26 13:11:13.166578441 (501h42m13.037661476s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166928107 (501h42m13.038011142s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx): protocol: read tcp 188.166.225.170:58818->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.166935010 (501h42m13.038018045s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167050671 (501h42m13.038133706s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: read tcp 188.166.225.170:41734->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.167056922 (501h42m13.038139977s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167110474 (501h42m13.038193519s) [seg_pos={0 0}, upload=440304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:11:13.167188423 (501h42m13.038271458s) [upload=440304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:11:13.167193292 (501h42m13.038276337s) [upload=440304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:11:13.455669712 (501h42m13.326752747s) [upload=440304]: Done uploading segments... Dec 26 13:11:13.455692565 (501h42m13.32677561s) [upload=440304]: Upload finished. err: <nil> Duration: 48.272892935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX] duration: 5.001741359s trace: https://jaeger.dev.storj.io/trace/4125c7bf904725df FAIL (2025-12-26 13:18:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001764484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X......] duration: 5.939049431s trace: https://jaeger.dev.storj.io/trace/4f70dfabd028a30c FAIL (2025-12-26 13:18:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ) context deadline exceeded ======LOGS====== Dec 26 13:13:44.039600490 (501h44m43.910683526s) [upload=440338]: Starting upload Dec 26 13:13:44.039706783 (501h44m43.910789818s) [upload=440338]: Done starting upload Dec 26 13:13:44.039812163 (501h44m43.910895209s) [upload=440338]: Uploading segments... Dec 26 13:13:44.040021552 (501h44m43.911104587s) [upload=440338]: Got next segment. Inline: false Dec 26 13:13:44.040076407 (501h44m43.911159442s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:13:44.040082468 (501h44m43.911165513s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:13:44.285063456 (501h44m44.156146501s) [seg_pos={0 0}, upload=440338]: Begin upload segment... Dec 26 13:13:44.285107319 (501h44m44.156190354s) [seg_pos={0 0}, upload=440338]: Done begin upload segment. Dec 26 13:13:44.285137938 (501h44m44.156220973s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285143979 (501h44m44.156227024s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285928403 (501h44m44.157011448s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286364825 (501h44m44.15744786s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286778732 (501h44m44.157861767s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288119686 (501h44m44.159202721s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288810491 (501h44m44.159893526s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.289561361 (501h44m44.160644406s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.291455989 (501h44m44.162539044s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293091623 (501h44m44.164174688s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293725951 (501h44m44.164808996s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295364963 (501h44m44.166447998s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295804639 (501h44m44.166887675s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297372034 (501h44m44.168455079s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297627641 (501h44m44.168710676s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.299020223 (501h44m44.170103258s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:45.853100559 (501h44m45.724183594s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 13:13:46.005209582 (501h44m45.876292627s) [upload=440338]: Next returned nil segment Dec 26 13:13:46.005226423 (501h44m45.876309459s) [upload=440338]: Waiting for error group managing segments... Dec 26 13:13:46.290650893 (501h44m46.161733948s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.294337503 (501h44m46.165420538s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.295158216 (501h44m46.166241251s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.302555681 (501h44m46.173638716s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.310665264 (501h44m46.181748299s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313398599 (501h44m46.184481654s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313991588 (501h44m46.185074623s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.322519997 (501h44m46.193603053s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.328469146 (501h44m46.199552191s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.550319338 (501h44m46.421402373s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.822512983 (501h44m46.693596018s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:47.538624588 (501h44m47.409707623s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431064312 (501h44m48.302147357s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431084371 (501h44m48.302167406s) [seg_pos={0 0}, upload=440338]: Segment reached optimal threshold of 13 pieces. Dec 26 13:13:48.431280053 (501h44m48.302363098s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 13:13:48.431288259 (501h44m48.302371294s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431384782 (501h44m48.302467837s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58472->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 13:13:48.431391416 (501h44m48.302474461s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431485244 (501h44m48.302568279s) [seg_pos={0 0}, upload=440338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:13:48.431552882 (501h44m48.302635917s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:13:48.431558793 (501h44m48.302641829s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:13:48.715618767 (501h44m48.586701812s) [upload=440338]: Done uploading segments... Dec 26 13:13:48.715642111 (501h44m48.586725146s) [upload=440338]: Upload finished. err: <nil> Duration: 5.939078225s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...X] duration: 5.001009516s trace: https://jaeger.dev.storj.io/trace/475832aeaa594d2f FAIL (2025-12-26 13:18:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001030445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..................X.XXX...X..X...................XX.X.......] duration: 57.564875094s trace: https://jaeger.dev.storj.io/trace/50ff20dcdf4be9b4 FAIL (2025-12-26 13:18:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (501h42m7.38824738s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517572723 (501h42m7.388655758s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517971292 (501h42m7.389054327s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518432961 (501h42m7.389516006s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518809027 (501h42m7.389892062s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519282127 (501h42m7.390365172s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519624399 (501h42m7.390707444s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519979325 (501h42m7.39106236s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.520359489 (501h42m7.391442534s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.521627654 (501h42m7.392710689s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.523533252 (501h42m7.394616287s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.524846152 (501h42m7.395929188s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.526350467 (501h42m7.397433542s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.528102293 (501h42m7.399185328s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.529588312 (501h42m7.400671347s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:08.839991300 (501h42m8.711074345s) [upload=440304]: Next returned nil segment Dec 26 13:11:08.840016208 (501h42m8.711099253s) [upload=440304]: Waiting for error group managing segments... Dec 26 13:11:09.181763785 (501h42m9.05284683s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.294980239 (501h42m9.166063284s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.435548195 (501h42m9.30663124s) [noise=true , address=145.79.227.28:4745 , piece_id=ICIDFSLXT62MLRKPE7NAJI4VOW6SLVMJL4KTNHHRJJK5XDAHBUWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.436737098 (501h42m9.307820143s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.439665284 (501h42m9.310748329s) [noise=true , address=145.79.227.40:4066 , piece_id=RPRP2QP7CPSCWSUI4DYVJ65TX5W2TLLBAHLFHJ7OSK4M62OXA6WA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.468255686 (501h42m9.339338731s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.758044865 (501h42m9.62912791s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.361420468 (501h42m10.232503513s) [noise=true , address=145.79.227.30:4790 , piece_id=6RSHKA5O3KNIPZG3EZD6KZZG7T4XFHRYYW7GBCSYE7G4JIWPBCNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.865755260 (501h42m10.736838295s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.384330849 (501h42m11.255413884s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.822065002 (501h42m11.693148047s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.921774887 (501h42m11.792857932s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:12.562307844 (501h42m12.433390889s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166097725 (501h42m13.03718077s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166117133 (501h42m13.037200178s) [seg_pos={0 0}, upload=440304]: Segment reached optimal threshold of 14 pieces. Dec 26 13:11:13.166391525 (501h42m13.03747457s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection Dec 26 13:11:13.166398408 (501h42m13.037481443s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166570326 (501h42m13.037653361s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d): protocol: expected piece hash; context canceled; EOF Dec 26 13:11:13.166578441 (501h42m13.037661476s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166928107 (501h42m13.038011142s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx): protocol: read tcp 188.166.225.170:58818->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.166935010 (501h42m13.038018045s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167050671 (501h42m13.038133706s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: read tcp 188.166.225.170:41734->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.167056922 (501h42m13.038139977s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167110474 (501h42m13.038193519s) [seg_pos={0 0}, upload=440304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:11:13.167188423 (501h42m13.038271458s) [upload=440304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:11:13.167193292 (501h42m13.038276337s) [upload=440304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:11:13.455669712 (501h42m13.326752747s) [upload=440304]: Done uploading segments... Dec 26 13:11:13.455692565 (501h42m13.32677561s) [upload=440304]: Upload finished. err: <nil> Duration: 57.564893611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [.X.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...] duration: 4.418990135s trace: https://jaeger.dev.storj.io/trace/48af4373e645d707 FAIL (2025-12-26 13:17:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.419012629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X.....] duration: 5.959701307s trace: https://jaeger.dev.storj.io/trace/4f70dfabd028a30c FAIL (2025-12-26 13:17:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ) context deadline exceeded ======LOGS====== Dec 26 13:13:44.039600490 (501h44m43.910683526s) [upload=440338]: Starting upload Dec 26 13:13:44.039706783 (501h44m43.910789818s) [upload=440338]: Done starting upload Dec 26 13:13:44.039812163 (501h44m43.910895209s) [upload=440338]: Uploading segments... Dec 26 13:13:44.040021552 (501h44m43.911104587s) [upload=440338]: Got next segment. Inline: false Dec 26 13:13:44.040076407 (501h44m43.911159442s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:13:44.040082468 (501h44m43.911165513s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:13:44.285063456 (501h44m44.156146501s) [seg_pos={0 0}, upload=440338]: Begin upload segment... Dec 26 13:13:44.285107319 (501h44m44.156190354s) [seg_pos={0 0}, upload=440338]: Done begin upload segment. Dec 26 13:13:44.285137938 (501h44m44.156220973s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285143979 (501h44m44.156227024s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285928403 (501h44m44.157011448s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286364825 (501h44m44.15744786s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286778732 (501h44m44.157861767s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288119686 (501h44m44.159202721s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288810491 (501h44m44.159893526s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.289561361 (501h44m44.160644406s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.291455989 (501h44m44.162539044s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293091623 (501h44m44.164174688s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293725951 (501h44m44.164808996s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295364963 (501h44m44.166447998s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295804639 (501h44m44.166887675s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297372034 (501h44m44.168455079s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297627641 (501h44m44.168710676s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.299020223 (501h44m44.170103258s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:45.853100559 (501h44m45.724183594s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 13:13:46.005209582 (501h44m45.876292627s) [upload=440338]: Next returned nil segment Dec 26 13:13:46.005226423 (501h44m45.876309459s) [upload=440338]: Waiting for error group managing segments... Dec 26 13:13:46.290650893 (501h44m46.161733948s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.294337503 (501h44m46.165420538s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.295158216 (501h44m46.166241251s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.302555681 (501h44m46.173638716s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.310665264 (501h44m46.181748299s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313398599 (501h44m46.184481654s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313991588 (501h44m46.185074623s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.322519997 (501h44m46.193603053s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.328469146 (501h44m46.199552191s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.550319338 (501h44m46.421402373s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.822512983 (501h44m46.693596018s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:47.538624588 (501h44m47.409707623s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431064312 (501h44m48.302147357s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431084371 (501h44m48.302167406s) [seg_pos={0 0}, upload=440338]: Segment reached optimal threshold of 13 pieces. Dec 26 13:13:48.431280053 (501h44m48.302363098s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 13:13:48.431288259 (501h44m48.302371294s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431384782 (501h44m48.302467837s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58472->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 13:13:48.431391416 (501h44m48.302474461s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431485244 (501h44m48.302568279s) [seg_pos={0 0}, upload=440338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:13:48.431552882 (501h44m48.302635917s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:13:48.431558793 (501h44m48.302641829s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:13:48.715618767 (501h44m48.586701812s) [upload=440338]: Done uploading segments... Dec 26 13:13:48.715642111 (501h44m48.586725146s) [upload=440338]: Upload finished. err: <nil> Duration: 5.95973411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................X.XXX...X..X...................XX.X......] duration: 44.517012003s trace: https://jaeger.dev.storj.io/trace/50ff20dcdf4be9b4 FAIL (2025-12-26 13:17:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (501h42m7.38824738s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517572723 (501h42m7.388655758s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517971292 (501h42m7.389054327s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518432961 (501h42m7.389516006s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518809027 (501h42m7.389892062s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519282127 (501h42m7.390365172s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519624399 (501h42m7.390707444s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519979325 (501h42m7.39106236s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.520359489 (501h42m7.391442534s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.521627654 (501h42m7.392710689s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.523533252 (501h42m7.394616287s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.524846152 (501h42m7.395929188s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.526350467 (501h42m7.397433542s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.528102293 (501h42m7.399185328s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.529588312 (501h42m7.400671347s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:08.839991300 (501h42m8.711074345s) [upload=440304]: Next returned nil segment Dec 26 13:11:08.840016208 (501h42m8.711099253s) [upload=440304]: Waiting for error group managing segments... Dec 26 13:11:09.181763785 (501h42m9.05284683s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.294980239 (501h42m9.166063284s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.435548195 (501h42m9.30663124s) [noise=true , address=145.79.227.28:4745 , piece_id=ICIDFSLXT62MLRKPE7NAJI4VOW6SLVMJL4KTNHHRJJK5XDAHBUWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.436737098 (501h42m9.307820143s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.439665284 (501h42m9.310748329s) [noise=true , address=145.79.227.40:4066 , piece_id=RPRP2QP7CPSCWSUI4DYVJ65TX5W2TLLBAHLFHJ7OSK4M62OXA6WA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.468255686 (501h42m9.339338731s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.758044865 (501h42m9.62912791s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.361420468 (501h42m10.232503513s) [noise=true , address=145.79.227.30:4790 , piece_id=6RSHKA5O3KNIPZG3EZD6KZZG7T4XFHRYYW7GBCSYE7G4JIWPBCNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.865755260 (501h42m10.736838295s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.384330849 (501h42m11.255413884s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.822065002 (501h42m11.693148047s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.921774887 (501h42m11.792857932s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:12.562307844 (501h42m12.433390889s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166097725 (501h42m13.03718077s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166117133 (501h42m13.037200178s) [seg_pos={0 0}, upload=440304]: Segment reached optimal threshold of 14 pieces. Dec 26 13:11:13.166391525 (501h42m13.03747457s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection Dec 26 13:11:13.166398408 (501h42m13.037481443s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166570326 (501h42m13.037653361s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d): protocol: expected piece hash; context canceled; EOF Dec 26 13:11:13.166578441 (501h42m13.037661476s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166928107 (501h42m13.038011142s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx): protocol: read tcp 188.166.225.170:58818->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.166935010 (501h42m13.038018045s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167050671 (501h42m13.038133706s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: read tcp 188.166.225.170:41734->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.167056922 (501h42m13.038139977s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167110474 (501h42m13.038193519s) [seg_pos={0 0}, upload=440304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:11:13.167188423 (501h42m13.038271458s) [upload=440304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:11:13.167193292 (501h42m13.038276337s) [upload=440304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:11:13.455669712 (501h42m13.326752747s) [upload=440304]: Done uploading segments... Dec 26 13:11:13.455692565 (501h42m13.32677561s) [upload=440304]: Upload finished. err: <nil> Duration: 44.517032172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..X.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X..] duration: 5.000582162s trace: https://jaeger.dev.storj.io/trace/48af4373e645d707 FAIL (2025-12-26 13:16:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000606047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X....] duration: 5.778088191s trace: https://jaeger.dev.storj.io/trace/4f70dfabd028a30c FAIL (2025-12-26 13:16:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ) context deadline exceeded ======LOGS====== Dec 26 13:13:44.039600490 (501h44m43.910683526s) [upload=440338]: Starting upload Dec 26 13:13:44.039706783 (501h44m43.910789818s) [upload=440338]: Done starting upload Dec 26 13:13:44.039812163 (501h44m43.910895209s) [upload=440338]: Uploading segments... Dec 26 13:13:44.040021552 (501h44m43.911104587s) [upload=440338]: Got next segment. Inline: false Dec 26 13:13:44.040076407 (501h44m43.911159442s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:13:44.040082468 (501h44m43.911165513s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:13:44.285063456 (501h44m44.156146501s) [seg_pos={0 0}, upload=440338]: Begin upload segment... Dec 26 13:13:44.285107319 (501h44m44.156190354s) [seg_pos={0 0}, upload=440338]: Done begin upload segment. Dec 26 13:13:44.285137938 (501h44m44.156220973s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285143979 (501h44m44.156227024s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285928403 (501h44m44.157011448s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286364825 (501h44m44.15744786s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286778732 (501h44m44.157861767s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288119686 (501h44m44.159202721s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288810491 (501h44m44.159893526s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.289561361 (501h44m44.160644406s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.291455989 (501h44m44.162539044s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293091623 (501h44m44.164174688s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293725951 (501h44m44.164808996s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295364963 (501h44m44.166447998s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295804639 (501h44m44.166887675s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297372034 (501h44m44.168455079s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297627641 (501h44m44.168710676s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.299020223 (501h44m44.170103258s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:45.853100559 (501h44m45.724183594s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 13:13:46.005209582 (501h44m45.876292627s) [upload=440338]: Next returned nil segment Dec 26 13:13:46.005226423 (501h44m45.876309459s) [upload=440338]: Waiting for error group managing segments... Dec 26 13:13:46.290650893 (501h44m46.161733948s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.294337503 (501h44m46.165420538s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.295158216 (501h44m46.166241251s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.302555681 (501h44m46.173638716s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.310665264 (501h44m46.181748299s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313398599 (501h44m46.184481654s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313991588 (501h44m46.185074623s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.322519997 (501h44m46.193603053s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.328469146 (501h44m46.199552191s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.550319338 (501h44m46.421402373s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.822512983 (501h44m46.693596018s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:47.538624588 (501h44m47.409707623s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431064312 (501h44m48.302147357s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431084371 (501h44m48.302167406s) [seg_pos={0 0}, upload=440338]: Segment reached optimal threshold of 13 pieces. Dec 26 13:13:48.431280053 (501h44m48.302363098s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 13:13:48.431288259 (501h44m48.302371294s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431384782 (501h44m48.302467837s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58472->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 13:13:48.431391416 (501h44m48.302474461s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431485244 (501h44m48.302568279s) [seg_pos={0 0}, upload=440338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:13:48.431552882 (501h44m48.302635917s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:13:48.431558793 (501h44m48.302641829s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:13:48.715618767 (501h44m48.586701812s) [upload=440338]: Done uploading segments... Dec 26 13:13:48.715642111 (501h44m48.586725146s) [upload=440338]: Upload finished. err: <nil> Duration: 5.778127337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [....................X.XXX...X..X...................XX.X.....] duration: 51.011579082s trace: https://jaeger.dev.storj.io/trace/50ff20dcdf4be9b4 FAIL (2025-12-26 13:16:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (501h42m7.38824738s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517572723 (501h42m7.388655758s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517971292 (501h42m7.389054327s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518432961 (501h42m7.389516006s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518809027 (501h42m7.389892062s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519282127 (501h42m7.390365172s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519624399 (501h42m7.390707444s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519979325 (501h42m7.39106236s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.520359489 (501h42m7.391442534s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.521627654 (501h42m7.392710689s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.523533252 (501h42m7.394616287s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.524846152 (501h42m7.395929188s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.526350467 (501h42m7.397433542s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.528102293 (501h42m7.399185328s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.529588312 (501h42m7.400671347s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:08.839991300 (501h42m8.711074345s) [upload=440304]: Next returned nil segment Dec 26 13:11:08.840016208 (501h42m8.711099253s) [upload=440304]: Waiting for error group managing segments... Dec 26 13:11:09.181763785 (501h42m9.05284683s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.294980239 (501h42m9.166063284s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.435548195 (501h42m9.30663124s) [noise=true , address=145.79.227.28:4745 , piece_id=ICIDFSLXT62MLRKPE7NAJI4VOW6SLVMJL4KTNHHRJJK5XDAHBUWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.436737098 (501h42m9.307820143s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.439665284 (501h42m9.310748329s) [noise=true , address=145.79.227.40:4066 , piece_id=RPRP2QP7CPSCWSUI4DYVJ65TX5W2TLLBAHLFHJ7OSK4M62OXA6WA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.468255686 (501h42m9.339338731s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.758044865 (501h42m9.62912791s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.361420468 (501h42m10.232503513s) [noise=true , address=145.79.227.30:4790 , piece_id=6RSHKA5O3KNIPZG3EZD6KZZG7T4XFHRYYW7GBCSYE7G4JIWPBCNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.865755260 (501h42m10.736838295s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.384330849 (501h42m11.255413884s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.822065002 (501h42m11.693148047s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.921774887 (501h42m11.792857932s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:12.562307844 (501h42m12.433390889s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166097725 (501h42m13.03718077s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166117133 (501h42m13.037200178s) [seg_pos={0 0}, upload=440304]: Segment reached optimal threshold of 14 pieces. Dec 26 13:11:13.166391525 (501h42m13.03747457s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection Dec 26 13:11:13.166398408 (501h42m13.037481443s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166570326 (501h42m13.037653361s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d): protocol: expected piece hash; context canceled; EOF Dec 26 13:11:13.166578441 (501h42m13.037661476s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166928107 (501h42m13.038011142s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx): protocol: read tcp 188.166.225.170:58818->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.166935010 (501h42m13.038018045s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167050671 (501h42m13.038133706s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: read tcp 188.166.225.170:41734->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.167056922 (501h42m13.038139977s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167110474 (501h42m13.038193519s) [seg_pos={0 0}, upload=440304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:11:13.167188423 (501h42m13.038271458s) [upload=440304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:11:13.167193292 (501h42m13.038276337s) [upload=440304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:11:13.455669712 (501h42m13.326752747s) [upload=440304]: Done uploading segments... Dec 26 13:11:13.455692565 (501h42m13.32677561s) [upload=440304]: Upload finished. err: <nil> Duration: 51.011593959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [...X.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X.] duration: 3.421892256s trace: https://jaeger.dev.storj.io/trace/48af4373e645d707 FAIL (2025-12-26 13:16:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.421916812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X...] duration: 6.128005287s trace: https://jaeger.dev.storj.io/trace/4f70dfabd028a30c FAIL (2025-12-26 13:15:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ) context deadline exceeded ======LOGS====== Dec 26 13:13:44.039600490 (501h44m43.910683526s) [upload=440338]: Starting upload Dec 26 13:13:44.039706783 (501h44m43.910789818s) [upload=440338]: Done starting upload Dec 26 13:13:44.039812163 (501h44m43.910895209s) [upload=440338]: Uploading segments... Dec 26 13:13:44.040021552 (501h44m43.911104587s) [upload=440338]: Got next segment. Inline: false Dec 26 13:13:44.040076407 (501h44m43.911159442s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:13:44.040082468 (501h44m43.911165513s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:13:44.285063456 (501h44m44.156146501s) [seg_pos={0 0}, upload=440338]: Begin upload segment... Dec 26 13:13:44.285107319 (501h44m44.156190354s) [seg_pos={0 0}, upload=440338]: Done begin upload segment. Dec 26 13:13:44.285137938 (501h44m44.156220973s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285143979 (501h44m44.156227024s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285928403 (501h44m44.157011448s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286364825 (501h44m44.15744786s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286778732 (501h44m44.157861767s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288119686 (501h44m44.159202721s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288810491 (501h44m44.159893526s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.289561361 (501h44m44.160644406s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.291455989 (501h44m44.162539044s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293091623 (501h44m44.164174688s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293725951 (501h44m44.164808996s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295364963 (501h44m44.166447998s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295804639 (501h44m44.166887675s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297372034 (501h44m44.168455079s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297627641 (501h44m44.168710676s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.299020223 (501h44m44.170103258s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:45.853100559 (501h44m45.724183594s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 13:13:46.005209582 (501h44m45.876292627s) [upload=440338]: Next returned nil segment Dec 26 13:13:46.005226423 (501h44m45.876309459s) [upload=440338]: Waiting for error group managing segments... Dec 26 13:13:46.290650893 (501h44m46.161733948s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.294337503 (501h44m46.165420538s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.295158216 (501h44m46.166241251s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.302555681 (501h44m46.173638716s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.310665264 (501h44m46.181748299s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313398599 (501h44m46.184481654s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313991588 (501h44m46.185074623s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.322519997 (501h44m46.193603053s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.328469146 (501h44m46.199552191s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.550319338 (501h44m46.421402373s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.822512983 (501h44m46.693596018s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:47.538624588 (501h44m47.409707623s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431064312 (501h44m48.302147357s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431084371 (501h44m48.302167406s) [seg_pos={0 0}, upload=440338]: Segment reached optimal threshold of 13 pieces. Dec 26 13:13:48.431280053 (501h44m48.302363098s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 13:13:48.431288259 (501h44m48.302371294s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431384782 (501h44m48.302467837s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58472->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 13:13:48.431391416 (501h44m48.302474461s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431485244 (501h44m48.302568279s) [seg_pos={0 0}, upload=440338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:13:48.431552882 (501h44m48.302635917s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:13:48.431558793 (501h44m48.302641829s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:13:48.715618767 (501h44m48.586701812s) [upload=440338]: Done uploading segments... Dec 26 13:13:48.715642111 (501h44m48.586725146s) [upload=440338]: Upload finished. err: <nil> Duration: 6.128037328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [X....................X.XXX...X..X...................XX.X....] duration: 50.769599117s trace: https://jaeger.dev.storj.io/trace/50ff20dcdf4be9b4 FAIL (2025-12-26 13:15:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (501h42m7.38824738s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517572723 (501h42m7.388655758s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517971292 (501h42m7.389054327s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518432961 (501h42m7.389516006s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518809027 (501h42m7.389892062s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519282127 (501h42m7.390365172s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519624399 (501h42m7.390707444s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519979325 (501h42m7.39106236s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.520359489 (501h42m7.391442534s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.521627654 (501h42m7.392710689s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.523533252 (501h42m7.394616287s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.524846152 (501h42m7.395929188s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.526350467 (501h42m7.397433542s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.528102293 (501h42m7.399185328s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.529588312 (501h42m7.400671347s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:08.839991300 (501h42m8.711074345s) [upload=440304]: Next returned nil segment Dec 26 13:11:08.840016208 (501h42m8.711099253s) [upload=440304]: Waiting for error group managing segments... Dec 26 13:11:09.181763785 (501h42m9.05284683s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.294980239 (501h42m9.166063284s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.435548195 (501h42m9.30663124s) [noise=true , address=145.79.227.28:4745 , piece_id=ICIDFSLXT62MLRKPE7NAJI4VOW6SLVMJL4KTNHHRJJK5XDAHBUWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.436737098 (501h42m9.307820143s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.439665284 (501h42m9.310748329s) [noise=true , address=145.79.227.40:4066 , piece_id=RPRP2QP7CPSCWSUI4DYVJ65TX5W2TLLBAHLFHJ7OSK4M62OXA6WA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.468255686 (501h42m9.339338731s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.758044865 (501h42m9.62912791s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.361420468 (501h42m10.232503513s) [noise=true , address=145.79.227.30:4790 , piece_id=6RSHKA5O3KNIPZG3EZD6KZZG7T4XFHRYYW7GBCSYE7G4JIWPBCNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.865755260 (501h42m10.736838295s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.384330849 (501h42m11.255413884s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.822065002 (501h42m11.693148047s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.921774887 (501h42m11.792857932s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:12.562307844 (501h42m12.433390889s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166097725 (501h42m13.03718077s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166117133 (501h42m13.037200178s) [seg_pos={0 0}, upload=440304]: Segment reached optimal threshold of 14 pieces. Dec 26 13:11:13.166391525 (501h42m13.03747457s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection Dec 26 13:11:13.166398408 (501h42m13.037481443s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166570326 (501h42m13.037653361s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d): protocol: expected piece hash; context canceled; EOF Dec 26 13:11:13.166578441 (501h42m13.037661476s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166928107 (501h42m13.038011142s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx): protocol: read tcp 188.166.225.170:58818->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.166935010 (501h42m13.038018045s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167050671 (501h42m13.038133706s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: read tcp 188.166.225.170:41734->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.167056922 (501h42m13.038139977s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167110474 (501h42m13.038193519s) [seg_pos={0 0}, upload=440304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:11:13.167188423 (501h42m13.038271458s) [upload=440304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:11:13.167193292 (501h42m13.038276337s) [upload=440304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:11:13.455669712 (501h42m13.326752747s) [upload=440304]: Done uploading segments... Dec 26 13:11:13.455692565 (501h42m13.32677561s) [upload=440304]: Upload finished. err: <nil> Duration: 50.769661126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [....X.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....X] duration: 5.00242888s trace: https://jaeger.dev.storj.io/trace/48af4373e645d707 FAIL (2025-12-26 13:15:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002455539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XX....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X..] duration: 5.970717714s trace: https://jaeger.dev.storj.io/trace/4f70dfabd028a30c FAIL (2025-12-26 13:15:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ) context deadline exceeded ======LOGS====== Dec 26 13:13:44.039600490 (501h44m43.910683526s) [upload=440338]: Starting upload Dec 26 13:13:44.039706783 (501h44m43.910789818s) [upload=440338]: Done starting upload Dec 26 13:13:44.039812163 (501h44m43.910895209s) [upload=440338]: Uploading segments... Dec 26 13:13:44.040021552 (501h44m43.911104587s) [upload=440338]: Got next segment. Inline: false Dec 26 13:13:44.040076407 (501h44m43.911159442s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:13:44.040082468 (501h44m43.911165513s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:13:44.285063456 (501h44m44.156146501s) [seg_pos={0 0}, upload=440338]: Begin upload segment... Dec 26 13:13:44.285107319 (501h44m44.156190354s) [seg_pos={0 0}, upload=440338]: Done begin upload segment. Dec 26 13:13:44.285137938 (501h44m44.156220973s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285143979 (501h44m44.156227024s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285928403 (501h44m44.157011448s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286364825 (501h44m44.15744786s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286778732 (501h44m44.157861767s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288119686 (501h44m44.159202721s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288810491 (501h44m44.159893526s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.289561361 (501h44m44.160644406s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.291455989 (501h44m44.162539044s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293091623 (501h44m44.164174688s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293725951 (501h44m44.164808996s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295364963 (501h44m44.166447998s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295804639 (501h44m44.166887675s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297372034 (501h44m44.168455079s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297627641 (501h44m44.168710676s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.299020223 (501h44m44.170103258s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:45.853100559 (501h44m45.724183594s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 13:13:46.005209582 (501h44m45.876292627s) [upload=440338]: Next returned nil segment Dec 26 13:13:46.005226423 (501h44m45.876309459s) [upload=440338]: Waiting for error group managing segments... Dec 26 13:13:46.290650893 (501h44m46.161733948s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.294337503 (501h44m46.165420538s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.295158216 (501h44m46.166241251s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.302555681 (501h44m46.173638716s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.310665264 (501h44m46.181748299s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313398599 (501h44m46.184481654s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313991588 (501h44m46.185074623s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.322519997 (501h44m46.193603053s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.328469146 (501h44m46.199552191s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.550319338 (501h44m46.421402373s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.822512983 (501h44m46.693596018s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:47.538624588 (501h44m47.409707623s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431064312 (501h44m48.302147357s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431084371 (501h44m48.302167406s) [seg_pos={0 0}, upload=440338]: Segment reached optimal threshold of 13 pieces. Dec 26 13:13:48.431280053 (501h44m48.302363098s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 13:13:48.431288259 (501h44m48.302371294s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431384782 (501h44m48.302467837s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58472->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 13:13:48.431391416 (501h44m48.302474461s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431485244 (501h44m48.302568279s) [seg_pos={0 0}, upload=440338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:13:48.431552882 (501h44m48.302635917s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:13:48.431558793 (501h44m48.302641829s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:13:48.715618767 (501h44m48.586701812s) [upload=440338]: Done uploading segments... Dec 26 13:13:48.715642111 (501h44m48.586725146s) [upload=440338]: Upload finished. err: <nil> Duration: 5.970748152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.....X.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX.....] duration: 5.001344162s trace: https://jaeger.dev.storj.io/trace/794f0c5ccb82aadf FAIL (2025-12-26 13:15:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001372495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....XX....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X.] duration: 6.136274132s trace: https://jaeger.dev.storj.io/trace/4f70dfabd028a30c FAIL (2025-12-26 13:14:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ) context deadline exceeded ======LOGS====== Dec 26 13:13:44.039600490 (501h44m43.910683526s) [upload=440338]: Starting upload Dec 26 13:13:44.039706783 (501h44m43.910789818s) [upload=440338]: Done starting upload Dec 26 13:13:44.039812163 (501h44m43.910895209s) [upload=440338]: Uploading segments... Dec 26 13:13:44.040021552 (501h44m43.911104587s) [upload=440338]: Got next segment. Inline: false Dec 26 13:13:44.040076407 (501h44m43.911159442s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:13:44.040082468 (501h44m43.911165513s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:13:44.285063456 (501h44m44.156146501s) [seg_pos={0 0}, upload=440338]: Begin upload segment... Dec 26 13:13:44.285107319 (501h44m44.156190354s) [seg_pos={0 0}, upload=440338]: Done begin upload segment. Dec 26 13:13:44.285137938 (501h44m44.156220973s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285143979 (501h44m44.156227024s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285928403 (501h44m44.157011448s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286364825 (501h44m44.15744786s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286778732 (501h44m44.157861767s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288119686 (501h44m44.159202721s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288810491 (501h44m44.159893526s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.289561361 (501h44m44.160644406s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.291455989 (501h44m44.162539044s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293091623 (501h44m44.164174688s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293725951 (501h44m44.164808996s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295364963 (501h44m44.166447998s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295804639 (501h44m44.166887675s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297372034 (501h44m44.168455079s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297627641 (501h44m44.168710676s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.299020223 (501h44m44.170103258s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:45.853100559 (501h44m45.724183594s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 13:13:46.005209582 (501h44m45.876292627s) [upload=440338]: Next returned nil segment Dec 26 13:13:46.005226423 (501h44m45.876309459s) [upload=440338]: Waiting for error group managing segments... Dec 26 13:13:46.290650893 (501h44m46.161733948s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.294337503 (501h44m46.165420538s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.295158216 (501h44m46.166241251s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.302555681 (501h44m46.173638716s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.310665264 (501h44m46.181748299s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313398599 (501h44m46.184481654s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313991588 (501h44m46.185074623s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.322519997 (501h44m46.193603053s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.328469146 (501h44m46.199552191s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.550319338 (501h44m46.421402373s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.822512983 (501h44m46.693596018s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:47.538624588 (501h44m47.409707623s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431064312 (501h44m48.302147357s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431084371 (501h44m48.302167406s) [seg_pos={0 0}, upload=440338]: Segment reached optimal threshold of 13 pieces. Dec 26 13:13:48.431280053 (501h44m48.302363098s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 13:13:48.431288259 (501h44m48.302371294s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431384782 (501h44m48.302467837s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58472->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 13:13:48.431391416 (501h44m48.302474461s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431485244 (501h44m48.302568279s) [seg_pos={0 0}, upload=440338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:13:48.431552882 (501h44m48.302635917s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:13:48.431558793 (501h44m48.302641829s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:13:48.715618767 (501h44m48.586701812s) [upload=440338]: Done uploading segments... Dec 26 13:13:48.715642111 (501h44m48.586725146s) [upload=440338]: Upload finished. err: <nil> Duration: 6.136303499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X....................X.XXX...X..X...................XX.X...] duration: 40.461376168s trace: https://jaeger.dev.storj.io/trace/50ff20dcdf4be9b4 FAIL (2025-12-26 13:14:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (501h42m7.38824738s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517572723 (501h42m7.388655758s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517971292 (501h42m7.389054327s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518432961 (501h42m7.389516006s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518809027 (501h42m7.389892062s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519282127 (501h42m7.390365172s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519624399 (501h42m7.390707444s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519979325 (501h42m7.39106236s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.520359489 (501h42m7.391442534s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.521627654 (501h42m7.392710689s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.523533252 (501h42m7.394616287s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.524846152 (501h42m7.395929188s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.526350467 (501h42m7.397433542s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.528102293 (501h42m7.399185328s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.529588312 (501h42m7.400671347s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:08.839991300 (501h42m8.711074345s) [upload=440304]: Next returned nil segment Dec 26 13:11:08.840016208 (501h42m8.711099253s) [upload=440304]: Waiting for error group managing segments... Dec 26 13:11:09.181763785 (501h42m9.05284683s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.294980239 (501h42m9.166063284s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.435548195 (501h42m9.30663124s) [noise=true , address=145.79.227.28:4745 , piece_id=ICIDFSLXT62MLRKPE7NAJI4VOW6SLVMJL4KTNHHRJJK5XDAHBUWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.436737098 (501h42m9.307820143s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.439665284 (501h42m9.310748329s) [noise=true , address=145.79.227.40:4066 , piece_id=RPRP2QP7CPSCWSUI4DYVJ65TX5W2TLLBAHLFHJ7OSK4M62OXA6WA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.468255686 (501h42m9.339338731s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.758044865 (501h42m9.62912791s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.361420468 (501h42m10.232503513s) [noise=true , address=145.79.227.30:4790 , piece_id=6RSHKA5O3KNIPZG3EZD6KZZG7T4XFHRYYW7GBCSYE7G4JIWPBCNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.865755260 (501h42m10.736838295s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.384330849 (501h42m11.255413884s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.822065002 (501h42m11.693148047s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.921774887 (501h42m11.792857932s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:12.562307844 (501h42m12.433390889s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166097725 (501h42m13.03718077s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166117133 (501h42m13.037200178s) [seg_pos={0 0}, upload=440304]: Segment reached optimal threshold of 14 pieces. Dec 26 13:11:13.166391525 (501h42m13.03747457s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection Dec 26 13:11:13.166398408 (501h42m13.037481443s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166570326 (501h42m13.037653361s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d): protocol: expected piece hash; context canceled; EOF Dec 26 13:11:13.166578441 (501h42m13.037661476s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166928107 (501h42m13.038011142s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx): protocol: read tcp 188.166.225.170:58818->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.166935010 (501h42m13.038018045s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167050671 (501h42m13.038133706s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: read tcp 188.166.225.170:41734->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.167056922 (501h42m13.038139977s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167110474 (501h42m13.038193519s) [seg_pos={0 0}, upload=440304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:11:13.167188423 (501h42m13.038271458s) [upload=440304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:11:13.167193292 (501h42m13.038276337s) [upload=440304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:11:13.455669712 (501h42m13.326752747s) [upload=440304]: Done uploading segments... Dec 26 13:11:13.455692565 (501h42m13.32677561s) [upload=440304]: Upload finished. err: <nil> Duration: 40.461407258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [......X.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX....] duration: 5.000664739s trace: https://jaeger.dev.storj.io/trace/794f0c5ccb82aadf FAIL (2025-12-26 13:14:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000696759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....XX....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....X] duration: 30.003095172s trace: https://jaeger.dev.storj.io/trace/4f70dfabd028a30c FAIL (2025-12-26 13:14:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ) context deadline exceeded ======LOGS====== Dec 26 13:13:44.039600490 (501h44m43.910683526s) [upload=440338]: Starting upload Dec 26 13:13:44.039706783 (501h44m43.910789818s) [upload=440338]: Done starting upload Dec 26 13:13:44.039812163 (501h44m43.910895209s) [upload=440338]: Uploading segments... Dec 26 13:13:44.040021552 (501h44m43.911104587s) [upload=440338]: Got next segment. Inline: false Dec 26 13:13:44.040076407 (501h44m43.911159442s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:13:44.040082468 (501h44m43.911165513s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:13:44.285063456 (501h44m44.156146501s) [seg_pos={0 0}, upload=440338]: Begin upload segment... Dec 26 13:13:44.285107319 (501h44m44.156190354s) [seg_pos={0 0}, upload=440338]: Done begin upload segment. Dec 26 13:13:44.285137938 (501h44m44.156220973s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285143979 (501h44m44.156227024s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.285928403 (501h44m44.157011448s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286364825 (501h44m44.15744786s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.286778732 (501h44m44.157861767s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288119686 (501h44m44.159202721s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.288810491 (501h44m44.159893526s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.289561361 (501h44m44.160644406s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.291455989 (501h44m44.162539044s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293091623 (501h44m44.164174688s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.293725951 (501h44m44.164808996s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295364963 (501h44m44.166447998s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.295804639 (501h44m44.166887675s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297372034 (501h44m44.168455079s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.297627641 (501h44m44.168710676s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:44.299020223 (501h44m44.170103258s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Uploading piece... Dec 26 13:13:45.853100559 (501h44m45.724183594s) [noise=true , address=77.90.31.11:4553 , piece_id=HKHLMTXCIHHFXBEK5KLNY426WRU2GGG5FEDM6AV7MM6FM37SK7RA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 13:13:46.005209582 (501h44m45.876292627s) [upload=440338]: Next returned nil segment Dec 26 13:13:46.005226423 (501h44m45.876309459s) [upload=440338]: Waiting for error group managing segments... Dec 26 13:13:46.290650893 (501h44m46.161733948s) [noise=true , address=145.79.227.29:4942 , piece_id=MOFCQNTLE6AFAERMPV5573V2KVLEVDMYLBC6GSFJLVY3I3V7GGGA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.294337503 (501h44m46.165420538s) [noise=true , address=145.79.227.14:4840 , piece_id=VCDCX5ZNAX47VJC7537XMSBIFBTDKDT4MUMYX5FBHDN6XP7OP7YQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.295158216 (501h44m46.166241251s) [noise=true , address=145.79.227.15:4873 , piece_id=ZINTP3CZ62HVB3TCUHF3WNXECIZVNIQDRLLTRY7TYTRAGCU6PXHA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.302555681 (501h44m46.173638716s) [noise=true , address=145.79.227.42:4121 , piece_id=DSPYXFTS5V2EX77M24ZSFFG6XBXALMVWJS4MGPK75ZKBE6NOTXPQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.310665264 (501h44m46.181748299s) [noise=true , address=145.79.227.3:4150 , piece_id=T6JBQYGCA37WMZTV5CJNLT6RBD5RDBG27RGYDLTTTMDWW5WDHFSQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313398599 (501h44m46.184481654s) [noise=true , address=145.79.227.6:4257 , piece_id=TFFTBS3DATZ4QX4WHOTTBGSIECOF6K6I2RJRXC4VE3TYAZ4K3Q5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.313991588 (501h44m46.185074623s) [noise=true , address=145.79.227.51:4410 , piece_id=JTSXQCZSDEX4ML4EKYHYKE2P4APD3EYWTBA6WIBQTAVPZ3JE3H2Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.322519997 (501h44m46.193603053s) [noise=true , address=145.79.227.13:4802 , piece_id=GEUDDQPSUHLDZCVJDJ2PGLUL4VINO7OU2KJ6C3LRC4NJNVFIBIWQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.328469146 (501h44m46.199552191s) [noise=true , address=145.79.227.52:4447 , piece_id=HRXOZ2BCVXPUZJOJ6KH5NSWVTWVZ3DEUYFZIJMF37C47OX3DKT3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.550319338 (501h44m46.421402373s) [noise=true , address=45.91.116.11:4120 , piece_id=O3VQO5BRJUH4D4MLBJYS7VYBRNRPUY7ZTU67ESICVZUREO2HC35A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:46.822512983 (501h44m46.693596018s) [noise=true , address=45.91.116.6:4976 , piece_id=YVE55LYFFMLNSAL4LUOHAQ47APZP5JIDVARA2IS4T7CBSQYD4NTQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:47.538624588 (501h44m47.409707623s) [noise=true , address=77.90.31.14:4661 , piece_id=5B3CLSMBMZABKLUGO22ULWN2VKC2GNGDOPLPGHCXK5TJ7IJ5RF5Q, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431064312 (501h44m48.302147357s) [noise=true , address=45.91.116.12:4227 , piece_id=LEBBAFINE6VMFOQQW4CCT6COHORW5X7DWSIQC5ZD5FBA2BHU3HDQ, seg_pos={0 0}, upload=440338]: Done uploading piece. err: <nil> Dec 26 13:13:48.431084371 (501h44m48.302167406s) [seg_pos={0 0}, upload=440338]: Segment reached optimal threshold of 13 pieces. Dec 26 13:13:48.431280053 (501h44m48.302363098s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 13:13:48.431288259 (501h44m48.302371294s) [noise=true , address=45.91.116.5:4298 , piece_id=5LFA2QK4DDKIDSYLOG3HXVFNI6M6AZGNMZDU437TYDHZSPKWYRTA, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431384782 (501h44m48.302467837s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58472->77.90.31.15:4697: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 13:13:48.431391416 (501h44m48.302474461s) [noise=true , address=77.90.31.15:4697 , piece_id=23EEJIX7ZB4LE7RL27XHT67DOLYCYHV6A6WVFL7FDX3V43LJDC3A, seg_pos={0 0}, upload=440338]: Long tail cancellation due to optimal threshold Dec 26 13:13:48.431485244 (501h44m48.302568279s) [seg_pos={0 0}, upload=440338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:13:48.431552882 (501h44m48.302635917s) [upload=440338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:13:48.431558793 (501h44m48.302641829s) [upload=440338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:13:48.715618767 (501h44m48.586701812s) [upload=440338]: Done uploading segments... Dec 26 13:13:48.715642111 (501h44m48.586725146s) [upload=440338]: Upload finished. err: <nil> Duration: 30.003118126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X....................X.XXX...X..X...................XX.X..] duration: 49.340988118s trace: https://jaeger.dev.storj.io/trace/50ff20dcdf4be9b4 FAIL (2025-12-26 13:13:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (501h42m7.38824738s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517572723 (501h42m7.388655758s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517971292 (501h42m7.389054327s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518432961 (501h42m7.389516006s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518809027 (501h42m7.389892062s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519282127 (501h42m7.390365172s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519624399 (501h42m7.390707444s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519979325 (501h42m7.39106236s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.520359489 (501h42m7.391442534s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.521627654 (501h42m7.392710689s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.523533252 (501h42m7.394616287s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.524846152 (501h42m7.395929188s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.526350467 (501h42m7.397433542s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.528102293 (501h42m7.399185328s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.529588312 (501h42m7.400671347s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:08.839991300 (501h42m8.711074345s) [upload=440304]: Next returned nil segment Dec 26 13:11:08.840016208 (501h42m8.711099253s) [upload=440304]: Waiting for error group managing segments... Dec 26 13:11:09.181763785 (501h42m9.05284683s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.294980239 (501h42m9.166063284s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.435548195 (501h42m9.30663124s) [noise=true , address=145.79.227.28:4745 , piece_id=ICIDFSLXT62MLRKPE7NAJI4VOW6SLVMJL4KTNHHRJJK5XDAHBUWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.436737098 (501h42m9.307820143s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.439665284 (501h42m9.310748329s) [noise=true , address=145.79.227.40:4066 , piece_id=RPRP2QP7CPSCWSUI4DYVJ65TX5W2TLLBAHLFHJ7OSK4M62OXA6WA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.468255686 (501h42m9.339338731s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.758044865 (501h42m9.62912791s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.361420468 (501h42m10.232503513s) [noise=true , address=145.79.227.30:4790 , piece_id=6RSHKA5O3KNIPZG3EZD6KZZG7T4XFHRYYW7GBCSYE7G4JIWPBCNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.865755260 (501h42m10.736838295s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.384330849 (501h42m11.255413884s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.822065002 (501h42m11.693148047s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.921774887 (501h42m11.792857932s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:12.562307844 (501h42m12.433390889s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166097725 (501h42m13.03718077s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166117133 (501h42m13.037200178s) [seg_pos={0 0}, upload=440304]: Segment reached optimal threshold of 14 pieces. Dec 26 13:11:13.166391525 (501h42m13.03747457s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection Dec 26 13:11:13.166398408 (501h42m13.037481443s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166570326 (501h42m13.037653361s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d): protocol: expected piece hash; context canceled; EOF Dec 26 13:11:13.166578441 (501h42m13.037661476s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166928107 (501h42m13.038011142s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx): protocol: read tcp 188.166.225.170:58818->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.166935010 (501h42m13.038018045s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167050671 (501h42m13.038133706s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: read tcp 188.166.225.170:41734->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.167056922 (501h42m13.038139977s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167110474 (501h42m13.038193519s) [seg_pos={0 0}, upload=440304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:11:13.167188423 (501h42m13.038271458s) [upload=440304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:11:13.167193292 (501h42m13.038276337s) [upload=440304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:11:13.455669712 (501h42m13.326752747s) [upload=440304]: Done uploading segments... Dec 26 13:11:13.455692565 (501h42m13.32677561s) [upload=440304]: Upload finished. err: <nil> Duration: 49.341005472s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.......X.XX......XXX..XXXXXXX.XXXX.X..X..X....XXXXXX...XX...] duration: 5.000642827s trace: https://jaeger.dev.storj.io/trace/794f0c5ccb82aadf FAIL (2025-12-26 13:13:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000670379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX....XX....X..X......X.X..XX.....X..XXXXXX..X....X..X.X....] duration: 6.188945507s trace: https://jaeger.dev.storj.io/trace/19317b6230fc38aa FAIL (2025-12-26 13:13:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: EMZDNU2YHW5J4L6QLUVFQMLYDYSONNOVMQWSASM3SX4YOVDTBS2A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: J3Y34TD6GHO34D47OLY6ESDBE5PKESEVW72FDHJQ6FVAVDIPEYBQ) context deadline exceeded ======LOGS====== Dec 26 13:08:57.993090971 (501h39m57.864174006s) [upload=440273]: Starting upload Dec 26 13:08:57.993142380 (501h39m57.864225425s) [upload=440273]: Done starting upload Dec 26 13:08:57.993247259 (501h39m57.864330304s) [upload=440273]: Uploading segments... Dec 26 13:08:57.993337971 (501h39m57.864421006s) [upload=440273]: Got next segment. Inline: false Dec 26 13:08:57.993378829 (501h39m57.864461874s) [upload=440273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:08:57.993385833 (501h39m57.864468878s) [upload=440273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:08:58.238403029 (501h39m58.109486074s) [seg_pos={0 0}, upload=440273]: Begin upload segment... Dec 26 13:08:58.238531252 (501h39m58.109614287s) [seg_pos={0 0}, upload=440273]: Done begin upload segment. Dec 26 13:08:58.238635120 (501h39m58.109718165s) [noise=true , address=77.90.31.11:4555 , piece_id=AQP5DZPGAIEZPV273FQVAG4RMM44XJOE22KQILDCD5TAS5EKTS5A, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.238703480 (501h39m58.109786525s) [noise=true , address=145.79.227.48:4303 , piece_id=MSF3CIU54LWHRSDTW3Z7SVDR3IWKMJQT3J5BYAXXQLXRZPWMXR2A, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.239586763 (501h39m58.110669808s) [noise=true , address=145.79.227.3:4150 , piece_id=AGMPYSBWMXBXNEC5GKDBQ2EQHBI4G7B6FSYI2HPMCSFEXJUE2QKA, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.239686812 (501h39m58.110769857s) [noise=true , address=45.91.116.6:4976 , piece_id=HRKZXKSGCL56LMGRFGNBJQZTRG4CLEDAPMYQNBCQLZL6AKPARZXQ, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.241489796 (501h39m58.112572871s) [noise=true , address=145.79.227.39:4013 , piece_id=ALZO2VDRN4SXN2CVA5JV5QXW7NOLJWTGIUGMVOLMDNBJSGDAMDPQ, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.242634025 (501h39m58.11371709s) [noise=true , address=45.91.116.12:4226 , piece_id=MO3QDRDV5OMF4FBWBVVLZDGQ5SOINW6XYRQRQ5CDRNQIUHLD52EA, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.243555440 (501h39m58.114638505s) [noise=true , address=145.79.227.35:4482 , piece_id=SZSO5A7VGUYPQ2CN5FGETZRBHSG327KLLGV56OCCK6GA6OBF5IVQ, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.243857614 (501h39m58.114940659s) [noise=true , address=145.79.227.21:4475 , piece_id=AY36P7IGDW6SEQNFLDX7WZRQGXTNZYTIZIWVEIJ5VD45YV56IGQQ, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.244691502 (501h39m58.115774547s) [noise=true , address=145.79.227.49:4338 , piece_id=PO66MGGDP4G4EXLIAQNNVCVETEV5LRKCMIU7S4YHPMHM4EMFJILA, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.245489111 (501h39m58.116572146s) [noise=true , address=145.79.227.15:4873 , piece_id=N7X4HJYWRIMCDWQBVRFQFDZF4X56WC6CUGPLTAKFW25W5PNDSM5Q, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.245954998 (501h39m58.117038033s) [noise=true , address=145.79.227.2:4114 , piece_id=NSYPPSRXG3PL65XTCDYD2FDDIZTG4M5EKELI3EGJME2APTECTJPQ, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.246874359 (501h39m58.117957404s) [noise=true , address=45.91.116.5:4297 , piece_id=HSBBSI5IDYE7A53L3FXKVXKOYJ2JHCGGSPZTT6KYNYU2TQQATS7Q, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.247613176 (501h39m58.118696211s) [noise=true , address=77.90.31.10:4518 , piece_id=TTODRMQU6OHKEEXBOSOVOQL6PR2UMHE7KZIGV4OLEGIYGUVREQLQ, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.248116525 (501h39m58.11919956s) [noise=true , address=45.91.116.14:4190 , piece_id=J3Y34TD6GHO34D47OLY6ESDBE5PKESEVW72FDHJQ6FVAVDIPEYBQ, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.249057597 (501h39m58.120140642s) [noise=true , address=145.79.227.4:4186 , piece_id=MBKISLXGP6PODYUIFRX3GOFRSMRPFBICBZP6C26MFQSCJB4LLABA, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:58.249514517 (501h39m58.120597562s) [noise=true , address=45.91.116.8:4046 , piece_id=EMZDNU2YHW5J4L6QLUVFQMLYDYSONNOVMQWSASM3SX4YOVDTBS2A, seg_pos={0 0}, upload=440273]: Uploading piece... Dec 26 13:08:59.951423490 (501h39m59.822506535s) [upload=440273]: Next returned nil segment Dec 26 13:08:59.951438359 (501h39m59.822521394s) [upload=440273]: Waiting for error group managing segments... Dec 26 13:09:00.245497579 (501h40m0.116580624s) [noise=true , address=145.79.227.4:4186 , piece_id=MBKISLXGP6PODYUIFRX3GOFRSMRPFBICBZP6C26MFQSCJB4LLABA, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.249694451 (501h40m0.120777496s) [noise=true , address=145.79.227.3:4150 , piece_id=AGMPYSBWMXBXNEC5GKDBQ2EQHBI4G7B6FSYI2HPMCSFEXJUE2QKA, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.250800426 (501h40m0.121883471s) [noise=true , address=145.79.227.2:4114 , piece_id=NSYPPSRXG3PL65XTCDYD2FDDIZTG4M5EKELI3EGJME2APTECTJPQ, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.263621831 (501h40m0.134704866s) [noise=true , address=145.79.227.15:4873 , piece_id=N7X4HJYWRIMCDWQBVRFQFDZF4X56WC6CUGPLTAKFW25W5PNDSM5Q, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.267138788 (501h40m0.138221853s) [noise=true , address=145.79.227.21:4475 , piece_id=AY36P7IGDW6SEQNFLDX7WZRQGXTNZYTIZIWVEIJ5VD45YV56IGQQ, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.268034222 (501h40m0.139117257s) [noise=true , address=145.79.227.35:4482 , piece_id=SZSO5A7VGUYPQ2CN5FGETZRBHSG327KLLGV56OCCK6GA6OBF5IVQ, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.269377621 (501h40m0.140460666s) [noise=true , address=145.79.227.48:4303 , piece_id=MSF3CIU54LWHRSDTW3Z7SVDR3IWKMJQT3J5BYAXXQLXRZPWMXR2A, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.274007847 (501h40m0.145090882s) [noise=true , address=145.79.227.39:4013 , piece_id=ALZO2VDRN4SXN2CVA5JV5QXW7NOLJWTGIUGMVOLMDNBJSGDAMDPQ, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.286829662 (501h40m0.157912697s) [noise=true , address=145.79.227.49:4338 , piece_id=PO66MGGDP4G4EXLIAQNNVCVETEV5LRKCMIU7S4YHPMHM4EMFJILA, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.505078145 (501h40m0.37616119s) [noise=true , address=45.91.116.8:4046 , piece_id=EMZDNU2YHW5J4L6QLUVFQMLYDYSONNOVMQWSASM3SX4YOVDTBS2A, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.633091145 (501h40m0.50417419s) [noise=true , address=45.91.116.12:4226 , piece_id=MO3QDRDV5OMF4FBWBVVLZDGQ5SOINW6XYRQRQ5CDRNQIUHLD52EA, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:00.850366826 (501h40m0.721449861s) [noise=true , address=77.90.31.10:4518 , piece_id=TTODRMQU6OHKEEXBOSOVOQL6PR2UMHE7KZIGV4OLEGIYGUVREQLQ, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:01.224857241 (501h40m1.095940286s) [noise=true , address=45.91.116.14:4190 , piece_id=J3Y34TD6GHO34D47OLY6ESDBE5PKESEVW72FDHJQ6FVAVDIPEYBQ, seg_pos={0 0}, upload=440273]: Done uploading piece. err: <nil> Dec 26 13:09:01.224888631 (501h40m1.095971676s) [seg_pos={0 0}, upload=440273]: Segment reached optimal threshold of 13 pieces. Dec 26 13:09:01.225280938 (501h40m1.096363984s) [noise=true , address=77.90.31.11:4555 , piece_id=AQP5DZPGAIEZPV273FQVAG4RMM44XJOE22KQILDCD5TAS5EKTS5A, seg_pos={0 0}, upload=440273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:32984->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:09:01.225292260 (501h40m1.096375325s) [noise=true , address=77.90.31.11:4555 , piece_id=AQP5DZPGAIEZPV273FQVAG4RMM44XJOE22KQILDCD5TAS5EKTS5A, seg_pos={0 0}, upload=440273]: Long tail cancellation due to optimal threshold Dec 26 13:09:01.225649720 (501h40m1.096732765s) [noise=true , address=45.91.116.6:4976 , piece_id=HRKZXKSGCL56LMGRFGNBJQZTRG4CLEDAPMYQNBCQLZL6AKPARZXQ, seg_pos={0 0}, upload=440273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 13:09:01.225660020 (501h40m1.096743065s) [noise=true , address=45.91.116.6:4976 , piece_id=HRKZXKSGCL56LMGRFGNBJQZTRG4CLEDAPMYQNBCQLZL6AKPARZXQ, seg_pos={0 0}, upload=440273]: Long tail cancellation due to optimal threshold Dec 26 13:09:01.226412583 (501h40m1.097495628s) [noise=true , address=45.91.116.5:4297 , piece_id=HSBBSI5IDYE7A53L3FXKVXKOYJ2JHCGGSPZTT6KYNYU2TQQATS7Q, seg_pos={0 0}, upload=440273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: read tcp 188.166.225.170:42504->45.91.116.5:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:09:01.226425408 (501h40m1.097508463s) [noise=true , address=45.91.116.5:4297 , piece_id=HSBBSI5IDYE7A53L3FXKVXKOYJ2JHCGGSPZTT6KYNYU2TQQATS7Q, seg_pos={0 0}, upload=440273]: Long tail cancellation due to optimal threshold Dec 26 13:09:01.226453561 (501h40m1.097536616s) [seg_pos={0 0}, upload=440273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:09:01.226565214 (501h40m1.097648259s) [upload=440273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:09:01.226574722 (501h40m1.097657767s) [upload=440273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:09:01.517898464 (501h40m1.388981509s) [upload=440273]: Done uploading segments... Dec 26 13:09:01.517920846 (501h40m1.389003891s) [upload=440273]: Upload finished. err: <nil> Duration: 6.188976447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X....................X.XXX...X..X...................XX.X.] duration: 52.831180986s trace: https://jaeger.dev.storj.io/trace/50ff20dcdf4be9b4 FAIL (2025-12-26 13:13:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (501h42m7.38824738s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517572723 (501h42m7.388655758s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.517971292 (501h42m7.389054327s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518432961 (501h42m7.389516006s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.518809027 (501h42m7.389892062s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519282127 (501h42m7.390365172s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519624399 (501h42m7.390707444s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.519979325 (501h42m7.39106236s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.520359489 (501h42m7.391442534s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.521627654 (501h42m7.392710689s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.523533252 (501h42m7.394616287s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.524846152 (501h42m7.395929188s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.526350467 (501h42m7.397433542s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.528102293 (501h42m7.399185328s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:07.529588312 (501h42m7.400671347s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Uploading piece... Dec 26 13:11:08.839991300 (501h42m8.711074345s) [upload=440304]: Next returned nil segment Dec 26 13:11:08.840016208 (501h42m8.711099253s) [upload=440304]: Waiting for error group managing segments... Dec 26 13:11:09.181763785 (501h42m9.05284683s) [noise=true , address=64.49.13.27:4960 , piece_id=VWNWXKCO2SSF32VIPK2BR2Q256ES5XTUDNDN2BKVNSBBT5UL6JPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.294980239 (501h42m9.166063284s) [noise=true , address=193.149.164.143:4697 , piece_id=JBTEGSUIU7BO3C4JFCY4NWOVFZC5HBPU7Y4MS6TACS7EFMQFQIEA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.435548195 (501h42m9.30663124s) [noise=true , address=145.79.227.28:4745 , piece_id=ICIDFSLXT62MLRKPE7NAJI4VOW6SLVMJL4KTNHHRJJK5XDAHBUWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.436737098 (501h42m9.307820143s) [noise=true , address=145.79.227.13:4819 , piece_id=AMYZ6KGGAZLOIRWPNOIYGQLEY3OC35YLTWPOTSAX56DDGMMJWFGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.439665284 (501h42m9.310748329s) [noise=true , address=145.79.227.40:4066 , piece_id=RPRP2QP7CPSCWSUI4DYVJ65TX5W2TLLBAHLFHJ7OSK4M62OXA6WA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.468255686 (501h42m9.339338731s) [noise=true , address=145.79.227.52:4460 , piece_id=GVA5CKPSUJCAPPYSFM57QY5QA3MQ2BROOGQIWGAPXDGCBZV2BJWQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:09.758044865 (501h42m9.62912791s) [noise=true , address=45.91.116.8:4056 , piece_id=N32TPT27JSJMOVB2IPNIHDDYXY3GGVKJUYRJREP7VLZWILUH52QQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.361420468 (501h42m10.232503513s) [noise=true , address=145.79.227.30:4790 , piece_id=6RSHKA5O3KNIPZG3EZD6KZZG7T4XFHRYYW7GBCSYE7G4JIWPBCNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:10.865755260 (501h42m10.736838295s) [noise=true , address=77.90.31.12:4605 , piece_id=JXML4B7HIRDO4IKXLPQUZFRSED5MVKAVTNV7BVEEX6AJVLUVF3EQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.384330849 (501h42m11.255413884s) [noise=true , address=145.79.227.44:4196 , piece_id=65XEAENNVRLOVLXUKEOCECLGJFB2AED2GEUUYTQI5BB6Z7O7HOJQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.822065002 (501h42m11.693148047s) [noise=true , address=77.90.31.16:4768 , piece_id=UISWRAGO7E2DKNYLGP2FNHSSFVNXPHR5ZODWDG3CUDH23YRRZL6A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:11.921774887 (501h42m11.792857932s) [noise=true , address=77.90.31.18:4838 , piece_id=GPF2GGWOVXJOO7GEQKR5PR4PZPDTZMO3XMGTSTLMYE6AJXZCEYFA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:12.562307844 (501h42m12.433390889s) [noise=true , address=77.90.31.11:4575 , piece_id=JZRG5AAS6J4WTDICJNK2DJPHSNRQL3NPBCZFW62OTGPKOX2RHYGA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166097725 (501h42m13.03718077s) [noise=true , address=77.90.31.15:4716 , piece_id=IPAOHNUCSIHV45SN5QMGZF5MWWTDLO3X3E6SIMDTTADVUL646V5A, seg_pos={0 0}, upload=440304]: Done uploading piece. err: <nil> Dec 26 13:11:13.166117133 (501h42m13.037200178s) [seg_pos={0 0}, upload=440304]: Segment reached optimal threshold of 14 pieces. Dec 26 13:11:13.166391525 (501h42m13.03747457s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->64.49.13.169:4709: use of closed network connection Dec 26 13:11:13.166398408 (501h42m13.037481443s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZDDUYBO4LB2BRKRHJCJFTQQEO456HCY53HAI7JOISLVH2NMUXNQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166570326 (501h42m13.037653361s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d): protocol: expected piece hash; context canceled; EOF Dec 26 13:11:13.166578441 (501h42m13.037661476s) [noise=true , address=77.90.31.14:4682 , piece_id=M4FNXD6PRKQLSJKEPLWNQ5RG7NM66USUOZEKMZPOVIEVJKINFXPQ, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.166928107 (501h42m13.038011142s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx): protocol: read tcp 188.166.225.170:58818->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.166935010 (501h42m13.038018045s) [noise=true , address=77.90.31.17:4804 , piece_id=NVGLQAN5SHE6KBPONVT6EAIEQP5EVSGLV7VEW7KUGGCSRLPEIHPA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167050671 (501h42m13.038133706s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: read tcp 188.166.225.170:41734->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:11:13.167056922 (501h42m13.038139977s) [noise=true , address=77.90.31.21:4897 , piece_id=KZEP3JD7BE446DBZPVPIWIQDOQQSM2OLGUMZX67JFWLPPAASS2OA, seg_pos={0 0}, upload=440304]: Long tail cancellation due to optimal threshold Dec 26 13:11:13.167110474 (501h42m13.038193519s) [seg_pos={0 0}, upload=440304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:11:13.167188423 (501h42m13.038271458s) [upload=440304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:11:13.167193292 (501h42m13.038276337s) [upload=440304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:11:13.455669712 (501h42m13.326752747s) [upload=440304]: Done uploading segments... Dec 26 13:11:13.455692565 (501h42m13.32677561s) [upload=440304]: Upload finished. err: <nil> Duration: 52.831210052s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)