Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 10.688840394s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:31:31 UTC) orbiter warning: context deadline exceeded Duration: 10.688867375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration: 6.463835234s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:31:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.463865261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.253297796s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:30:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.253326161s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X....XXX..X....................X............X.X.X......X...] duration: 3.46405802s trace: https://jaeger.dev.storj.io/trace/3555d8c9865f0020 FAIL (2025-12-25 15:30:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464082527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.....X...X..............................................] duration: 5.72743775s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:30:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.727457248s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 16.52610722s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:30:44 UTC) orbiter warning: context deadline exceeded Duration: 16.526138871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.25809614s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:30:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.258124324s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X....X................................................] duration: 6.439790264s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:30:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.439822405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.153692914s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:29:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 5.153736989s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 7.915803231s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:29:58 UTC) orbiter warning: context deadline exceeded Duration: 7.915829912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.X....XXX..X....................X............X.X.X......X..] duration: 3.566767667s trace: https://jaeger.dev.storj.io/trace/3555d8c9865f0020 FAIL (2025-12-25 15:29:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566796772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.....X...X.............................................] duration: 5.643586252s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:29:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.643617693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XX.X....XXX..X....................X............X.X.X......X.] duration: 3.524474193s trace: https://jaeger.dev.storj.io/trace/3555d8c9865f0020 FAIL (2025-12-25 15:29:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524498007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X.....X...X............................................] duration: 5.665360645s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:29:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.665486104s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XXX.X....XXX..X....................X............X.X.X......X] duration: 5.000761919s trace: https://jaeger.dev.storj.io/trace/3555d8c9865f0020 FAIL (2025-12-25 15:29:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00078923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration: 6.480647449s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.480673138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.649013842s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:28:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.649033338s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.795736602s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:28:50 UTC) orbiter warning: context deadline exceeded Duration: 7.795755549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X....X..............................................] duration: 6.404058952s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:28:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.404089319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 6.478944563s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:28:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.478966966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX....X.....X...X...........................................] duration: 5.802396593s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:28:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.802415168s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 14.429641003s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:28:01 UTC) orbiter warning: context deadline exceeded Duration: 14.429671341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XXXX.X....XXX..X....................X............X.X.X......] duration: 5.000669101s trace: https://jaeger.dev.storj.io/trace/401fd9f6a1534f79 FAIL (2025-12-25 15:28:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000697696s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 4.654656485s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:27:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.654686794s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 6.587821111s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:27:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.587838193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX....X.....X...X..........................................] duration: 5.804655247s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:27:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.804685444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration: 6.580879936s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:27:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.580907109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 3.706308394s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:27:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 3.706336279s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.146913193s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:27:03 UTC) orbiter warning: context deadline exceeded Duration: 6.146941748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XXXX.X....XXX..X....................X............X.X.X.....] duration: 3.469959167s trace: https://jaeger.dev.storj.io/trace/401fd9f6a1534f79 FAIL (2025-12-25 15:26:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.665341004s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:26:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.66537088s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X....X..........................................] duration: 6.477797318s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:26:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.477830482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XXXX.X....XXX..X....................X............X.X.X....] duration: 4.907495045s trace: https://jaeger.dev.storj.io/trace/401fd9f6a1534f79 FAIL (2025-12-25 15:26:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.907509572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX....X.....X...X.........................................] duration: 5.795132504s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:26:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.79518331s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.713789656s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:26:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.713813703s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 6.527631754s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:26:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.527660718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.51227938s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:26:01 UTC) orbiter warning: context deadline exceeded Duration: 6.512304177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.938379958s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:25:43 UTC) orbiter warning: context deadline exceeded Duration: 5.938403774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XXXX.X....XXX..X....................X............X.X.X...] duration: 4.021806315s trace: https://jaeger.dev.storj.io/trace/401fd9f6a1534f79 FAIL (2025-12-25 15:25:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.021844648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 4.093667223s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.093694805s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX....X.....X...X........................................] duration: 5.774783995s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:25:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.774814734s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 4.56033926s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:24:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 4.560367445s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 6.520801146s trace: https://jaeger.dev.storj.io/trace/648a4d9ea8398e98 FAIL (2025-12-25 15:24:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded ======LOGS====== LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece... Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments... Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil> Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces. Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments... Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil> Duration: 6.52082369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 8.876312661s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:24:44 UTC) orbiter warning: context deadline exceeded Duration: 8.876352377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....XXXX.X....XXX..X....................X............X.X.X..] duration: 4.071223766s trace: https://jaeger.dev.storj.io/trace/401fd9f6a1534f79 FAIL (2025-12-25 15:24:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.071246288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX....X.....X...X.......................................] duration: 5.780253991s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:24:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.780309537s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.662452068s trace: https://jaeger.dev.storj.io/trace/646704995e2fff8 FAIL (2025-12-25 15:24:08 UTC) orbiter warning: context deadline exceeded Duration: 5.662483327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.104653837s trace: https://jaeger.dev.storj.io/trace/1273039440c21470 FAIL (2025-12-25 15:24:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=420943]: Uploading piece... Dec 25 15:02:23.606060719 (479h33m23.477143764s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.606558357 (479h33m23.477641402s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607258420 (479h33m23.478341465s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.607625057 (479h33m23.478708103s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608073862 (479h33m23.479156897s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608320872 (479h33m23.479403917s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.608821035 (479h33m23.47990408s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609179337 (479h33m23.480262382s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609381032 (479h33m23.480464067s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:23.609866085 (479h33m23.48094913s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Uploading piece... Dec 25 15:02:24.427155891 (479h33m24.298238926s) [upload=420943]: Next returned nil segment Dec 25 15:02:24.427176210 (479h33m24.298259255s) [upload=420943]: Waiting for error group managing segments... Dec 25 15:02:24.821026256 (479h33m24.692109291s) [noise=true , address=185.167.204.206:4054 , piece_id=NGTR4L7IUCTBXRQKQ3ASX7BVTOJ2V3YDYVF4ALTLNYYZWQXIOWDQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821399076 (479h33m24.692482111s) [noise=true , address=185.167.204.205:4020 , piece_id=U4G7DFBTMMTNU2DLCDNMCEGWV5FRXWVEOTGU7LBHQCLA2ADWATHA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.821773199 (479h33m24.692856234s) [noise=true , address=185.167.204.206:4064 , piece_id=CIZV6UB4XXEHXCGGXLM25HKIMLRDAGXZ647HUEQNV3MTZTTWD6GA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.822149454 (479h33m24.693232489s) [noise=true , address=185.167.204.205:4031 , piece_id=JWOTQCQO2OY2O46SLPIHM2B3SWASCJPOSJ2CWRSWI4P4Y23P5Q7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.823698033 (479h33m24.694781079s) [noise=true , address=185.167.204.206:4049 , piece_id=RYIWPXL2S7HAECNBO4KALMARLZQEKOEWSTJMHXGFD2XTNV62VGRA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.824247891 (479h33m24.695330936s) [noise=true , address=185.167.204.205:4003 , piece_id=E5V6NFKTTNPJBLYSO4XFEVD23YTQCGOEXMOZZN4LRSDLAJEE7W5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825376539 (479h33m24.696459574s) [noise=true , address=185.167.204.206:4045 , piece_id=A4ZRDJ47V3LMXDV7QVWGGRKXVYCODWT3IA64U3DLAT42JW27QO5Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.825990509 (479h33m24.697073554s) [noise=true , address=185.167.204.206:4053 , piece_id=NI5V4SLYRPEJX3V4OHCI6NTFBNTI7XYXSUT4RMUB57HHWNJ3JKNQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.827193659 (479h33m24.698276704s) [noise=true , address=185.167.204.205:4001 , piece_id=NDYVLDBDRK43EFLZF3VKARH5NHYTWHRARWEKGFUCIOHDRUHIWTVQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.828931458 (479h33m24.700014494s) [noise=true , address=185.167.204.206:4046 , piece_id=ONNQBGCSUXFOS357AFS57XWSSUG64ZMO3X4Z3IZOGAIUWE7X6ZKQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.829906415 (479h33m24.70098947s) [noise=true , address=185.167.204.206:4071 , piece_id=NHGAORQBNMRUW3JMP7LBYWH4EYWAWEHRKNJAAQ5UCVODNDKPUJIA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.830652406 (479h33m24.701735441s) [noise=true , address=185.167.204.206:4048 , piece_id=6FJAKL2K7YRSOT3AXKOEM7U5RJIMU6XVXYKJ2YTSAU4LSQIUCSQA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.831199056 (479h33m24.702282111s) [noise=true , address=185.167.204.205:4029 , piece_id=MD6XNN6EXTP3TYMAKWM6TWQ2JKPR6KM52UM2EN67MZZC6BBJIROA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833160963 (479h33m24.704244008s) [noise=true , address=185.167.204.205:4033 , piece_id=MQTNO7QBGI25IX7BQ36WOKGRIVR5WEJMYYV5FOEFNX2IMLIHZ2GQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.833954233 (479h33m24.705037279s) [noise=true , address=185.167.204.205:4006 , piece_id=UOINPY4XALJL7MUX5HLZGZW4CSHREZP2D5L662LEAAWJO4GYONEQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.836340707 (479h33m24.707423752s) [noise=true , address=185.167.204.206:4059 , piece_id=ZCBQQWBTJY6JVPAJI5VUSTS6CTYFTM5BRWBXVJAYRFEVHADF3EAQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:24.847455291 (479h33m24.718538326s) [noise=true , address=185.167.204.205:4034 , piece_id=NWCLIRLPPDLRUXGMAYMJMOISXQO3W7L26UQWDPPMNUKBNOCZOQBQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.442368876 (479h33m25.313451921s) [noise=true , address=81.171.14.70:4035 , piece_id=MKILOKPD2KFHJC5H3CWX2VDFFZPJN7GGXM6TCDXQQFRZ5B7BAOYA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.470975298 (479h33m25.342058343s) [noise=true , address=81.171.14.70:4025 , piece_id=TWXQJO4W57HPVYRIPE3ZGUORDQY7NW23DAYXSXS2M7MDNV333LYQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.512195237 (479h33m25.383278282s) [noise=true , address=81.171.14.70:4026 , piece_id=KU6FW3MP56X7IGOZPY5PGOAV5IQGBVIW3ULI2L4ACCQWLYKHZUUQ, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.513876048 (479h33m25.384959083s) [noise=true , address=81.171.14.70:4032 , piece_id=J2HJNJQXVG2ATF6LGI54IHFBJDO4HM6KSTBYHTSMA6ZJZL6USXKA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.529308794 (479h33m25.400391839s) [noise=true , address=81.171.14.70:4038 , piece_id=JTBO4LN7SABOJS3D6WUES6XZXXONFKJBEQSNRZR5B4TXP2E4P76A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.532359544 (479h33m25.403442589s) [noise=true , address=81.171.14.70:4048 , piece_id=FOEQVNJ266EYNF6BOZOCGQGGA3VM7R3NAAUB62LKF4FRG2AXPKSA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:25.561767973 (479h33m25.432851018s) [noise=true , address=81.171.14.70:4029 , piece_id=ZEIRS2X3FCQYLE3CIOCNUSW26LJ4OT5Z5DOQBJCRZHZTL7WTJ2NA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.075442421 (479h33m25.946525456s) [noise=true , address=89.149.201.147:4013 , piece_id=GWJOI7HNLLBZ3RKJUCEJXYLYRASC7JGTHJH2F5RT7QR36ILA5MGA, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139345414 (479h33m26.010428459s) [noise=true , address=89.149.201.147:4007 , piece_id=R7LDBZTBETOPXJH7KMRJOVP67LACOGDGQUAN6ACBAVVOIRGKR36Q, seg_pos={0 0}, upload=420943]: Done uploading piece. err: <nil> Dec 25 15:02:26.139368788 (479h33m26.010451833s) [seg_pos={0 0}, upload=420943]: Segment reached optimal threshold of 26 pieces. Dec 25 15:02:26.139570672 (479h33m26.010653707s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139578778 (479h33m26.010661823s) [noise=true , address=89.149.201.147:4021 , piece_id=T2BTF5PGMNGE6GC35N6O6L4WZ7N4YVHVDA6U4JDAO7ADZQRPAH2A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139872858 (479h33m26.010955903s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF Dec 25 15:02:26.139881524 (479h33m26.010964559s) [noise=true , address=89.149.201.147:4005 , piece_id=GOPTZZ5IUTEGZN7YMHO3L5QQGZXY3NC2PJOGLFBC3UL4WD4S7E7A, seg_pos={0 0}, upload=420943]: Long tail cancellation due to optimal threshold Dec 25 15:02:26.139902044 (479h33m26.010985089s) [seg_pos={0 0}, upload=420943]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 25 15:02:26.139986594 (479h33m26.01106964s) [upload=420943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 15:02:26.139991914 (479h33m26.011074949s) [upload=420943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 15:02:26.368105170 (479h33m26.239188216s) [upload=420943]: Done uploading segments... Dec 25 15:02:26.368127482 (479h33m26.239210527s) [upload=420943]: Upload finished. err: <nil> Duration: 5.104694916s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX....X.....X...X......................................] duration: 5.798899686s trace: https://jaeger.dev.storj.io/trace/7e027e932241536e FAIL (2025-12-25 15:23:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded ======LOGS====== c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments... Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment... Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment. Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece... Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments... Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil> Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces. Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments... Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil> Duration: 5.798924694s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....XXXX.X....XXX..X....................X............X.X.X.] duration: 4.469982117s trace: https://jaeger.dev.storj.io/trace/401fd9f6a1534f79 FAIL (2025-12-25 15:23:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.470019619s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)