Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........X........X......X...................X.X.........X..] duration: 5.984294986s trace: https://jaeger.dev.storj.io/trace/5ff4d8b696fcf7dc FAIL (2025-12-08 10:24:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.984318441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..................X.............] duration: 5.902916194s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:24:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.902937004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.818548755s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:24:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.818588622s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.427182923s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:24:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.427222208s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.....X..............................X.....................] duration: 6.057270344s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:24:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.057303176s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X....................................X.....X.....] duration: 3.55745439s trace: https://jaeger.dev.storj.io/trace/a5556d780757d4 FAIL (2025-12-08 10:23:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.557484438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..................X............] duration: 5.835353712s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:23:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.835371897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 4.927952703s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:23:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.927986545s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.169483033s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:23:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.169511176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.....X..............................X....................] duration: 6.012570801s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:23:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.012591891s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........X........X......X...................X.X.........X.] duration: 5.57681248s trace: https://jaeger.dev.storj.io/trace/5ff4d8b696fcf7dc FAIL (2025-12-08 10:23:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.576829241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X....................................X.....X....] duration: 3.507205331s trace: https://jaeger.dev.storj.io/trace/a5556d780757d4 FAIL (2025-12-08 10:23:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507231071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........X........X......X...................X.X.........X] duration: 1m0.000792893s trace: https://jaeger.dev.storj.io/trace/5ff4d8b696fcf7dc FAIL (2025-12-08 10:23:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00090131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 4.931839219s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:23:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.931868505s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.504838142s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:23:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.5048684s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..................X...........] duration: 6.246379585s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:23:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.246424071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.....X..............................X...................] duration: 5.804147056s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:23:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.804194075s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.489040411s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:22:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.489066351s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..................X..........] duration: 6.823384823s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:22:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.823474202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X....................................X.....X...] duration: 3.537812903s trace: https://jaeger.dev.storj.io/trace/a5556d780757d4 FAIL (2025-12-08 10:22:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537843952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.036593795s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:22:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.036623591s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X....................................X.....X..] duration: 3.447898922s trace: https://jaeger.dev.storj.io/trace/a5556d780757d4 FAIL (2025-12-08 10:22:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447926586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.....X..............................X..................] duration: 6.238310865s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:22:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.238337436s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............X........X......X...................X.X.........] duration: 5.683489384s trace: https://jaeger.dev.storj.io/trace/76353e8535c77a45 FAIL (2025-12-08 10:22:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.683519992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.062661112s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:22:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.062693516s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............X........X......X...................X.X........] duration: 5.647022142s trace: https://jaeger.dev.storj.io/trace/76353e8535c77a45 FAIL (2025-12-08 10:21:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.6470524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X....................................X.....X.] duration: 3.502807357s trace: https://jaeger.dev.storj.io/trace/a5556d780757d4 FAIL (2025-12-08 10:21:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502835019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 9.002018322s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:21:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 9.002043018s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.415565241s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:21:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.415593855s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.061744802s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:21:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.061771311s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..................X.........] duration: 6.116714875s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:21:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.116740043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X....................................X.....X] duration: 5.000467495s trace: https://jaeger.dev.storj.io/trace/a5556d780757d4 FAIL (2025-12-08 10:21:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000491911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.....X..............................X.................] duration: 6.115336043s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:21:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.115361302s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.483766407s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:21:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.483790664s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.281733368s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:21:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.281800647s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X........X......X...................X.X.......] duration: 5.855042956s trace: https://jaeger.dev.storj.io/trace/76353e8535c77a45 FAIL (2025-12-08 10:20:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.855075226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X..................X........] duration: 6.021633396s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:20:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.021668952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.409021399s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:20:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.40904727s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X....................................X.....] duration: 3.50604824s trace: https://jaeger.dev.storj.io/trace/6754231f36d7ab1b FAIL (2025-12-08 10:20:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506277537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.....X..............................X................] duration: 7.0577604s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:20:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.057788813s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 8.487807517s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:20:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 8.487838498s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 4.99080923s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:20:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.990843115s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.416474491s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:20:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.416504227s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.409933397s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:19:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.409965578s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 8.742143753s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:19:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 8.742169041s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............X........X......X...................X.X......] duration: 5.917704113s trace: https://jaeger.dev.storj.io/trace/76353e8535c77a45 FAIL (2025-12-08 10:19:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.917744151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X..................X.......] duration: 7.4776274s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:19:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.47764824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.....X..............................X...............] duration: 7.49305077s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:19:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.493079626s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.950547226s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:19:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.95058016s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X....................................X....] duration: 3.511790876s trace: https://jaeger.dev.storj.io/trace/6754231f36d7ab1b FAIL (2025-12-08 10:19:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511821965s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)