Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..XX......X......X..X.....XXXX.X....XXX..X..................] duration: 3.968351s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:08:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.968376488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.67912406s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:08:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.679166722s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....X....................] duration: 6.124367995s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:08:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.124393835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................XX....X.....X...X.................] duration: 6.653088669s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:08:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 6.653118145s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...XX......X......X..X.....XXXX.X....XXX..X.................] duration: 3.55299892s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:07:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553021623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 4.668191605s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:07:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.668229657s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....X...................] duration: 6.273525278s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:07:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.273552649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................XX....X.....X...X................] duration: 5.797548517s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:07:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.797577984s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....XX......X......X..X.....XXXX.X....XXX..X................] duration: 4.303207705s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:07:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.303242942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 4.715739892s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:06:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.71581711s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....X..................] duration: 6.542565667s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:06:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.542592258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XX....X.....X...X...............] duration: 5.875604711s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:06:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.875641852s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X....X.................] duration: 6.460415509s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:06:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.460431449s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....XX......X......X..X.....XXXX.X....XXX..X...............] duration: 4.957409633s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:05:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.957452486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 4.754912155s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:05:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.754938685s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XX....X.....X...X..............] duration: 5.768253545s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:05:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.768280858s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [......XX......X......X..X.....XXXX.X....XXX..X..............] duration: 5.000919575s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:05:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000944252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.131392667s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:05:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 5.131424056s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XX....X.....X...X.............] duration: 5.804297057s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:05:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.804325471s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X....X................] duration: 6.206250653s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:04:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.206282272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 4.28569667s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:04:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.285731043s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X......XX......X......X..X.....XXXX.X....XXX..X.............] duration: 4.685814934s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:04:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.685841384s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X......XX......X......X..X.....XXXX.X....XXX..X............] duration: 3.529606748s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:04:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529638459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 4.486821424s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:04:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.48685024s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................XX....X.....X...X............] duration: 5.774453699s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:03:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.774488185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X....X...............] duration: 6.200605256s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:03:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.200637758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.654821326s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:03:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.654846655s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..X......XX......X......X..X.....XXXX.X....XXX..X...........] duration: 4.582973264s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:03:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.582998191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X....X..............] duration: 6.413041023s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.413063646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 4.300334783s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:02:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.30041195s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................XX....X.....X...X...........] duration: 5.668168997s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:02:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.668194955s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 20.085062488s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:02:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 20.085105059s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [...X......XX......X......X..X.....XXXX.X....XXX..X..........] duration: 4.953574821s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:02:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.953592807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X....X.............] duration: 39.310416191s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:02:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 39.310439225s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [....X......XX......X......X..X.....XXXX.X....XXX..X.........] duration: 4.74159739s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:02:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.741622618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................XX....X.....X...X..........] duration: 5.825812247s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:02:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.825847063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................XX....X.....X...X.........] duration: 6.051040299s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:01:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 6.051065127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X....X............] duration: 44.015501772s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:01:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 44.015543561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X....X......XX......X......X..X.....XXXX.X....XXX..X........] duration: 3.641232494s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 15:01:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.641261318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................XX....X.....X...X........] duration: 5.731387877s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:00:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.731414146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 47.650914343s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:00:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 47.650945973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................XX....X.....X...X.......] duration: 5.77214376s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:00:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.772178175s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X....X......XX......X......X..X.....XXXX.X....XXX..X.......] duration: 3.551136904s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 14:59:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551157554s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 13.242511565s trace: https://jaeger.dev.storj.io/trace/4a034410a446d28d FAIL (2025-12-25 14:59:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.242538007s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X....X......XX......X......X..X.....XXXX.X....XXX..X......] duration: 5.00142738s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 14:59:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001449672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XX....X.....X...X......] duration: 5.95512511s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 14:59:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.955147723s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 14.568584439s trace: https://jaeger.dev.storj.io/trace/4a034410a446d28d FAIL (2025-12-25 14:59:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.568608576s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...X....X......XX......X......X..X.....XXXX.X....XXX..X.....] duration: 3.508555835s trace: https://jaeger.dev.storj.io/trace/77aa25dfa3c5b5ee FAIL (2025-12-25 14:59:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508581102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X....X..........] duration: 42.070049587s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 14:59:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 42.070066198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................XX....X.....X...X.....] duration: 5.811646581s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 14:59:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.811672401s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)