Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.500535525s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:17:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.500558017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.....X..............................X............] duration: 6.04683201s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:17:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.04685801s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..................X....] duration: 6.002208879s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:17:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.00224074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................X........X......X...................X.X...] duration: 5.743461889s trace: https://jaeger.dev.storj.io/trace/76353e8535c77a45 FAIL (2025-12-08 10:17:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.743541008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.909622006s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:17:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.909654657s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X....................................X.] duration: 3.62494175s trace: https://jaeger.dev.storj.io/trace/6754231f36d7ab1b FAIL (2025-12-08 10:17:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.624969202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................X........X......X...................X.X..] duration: 6.72270805s trace: https://jaeger.dev.storj.io/trace/76353e8535c77a45 FAIL (2025-12-08 10:17:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.722740912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..................X...] duration: 7.002117897s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:17:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.002152892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.....X..............................X...........] duration: 5.786372053s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:16:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.786417641s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.499457708s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:16:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.499475272s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..................X..] duration: 6.906350046s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:16:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.906380613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X....................................X] duration: 5.001179005s trace: https://jaeger.dev.storj.io/trace/6754231f36d7ab1b FAIL (2025-12-08 10:16:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001204032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 7.3025251s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:16:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.302555078s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.31422524s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:16:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.314256328s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.448589836s trace: https://jaeger.dev.storj.io/trace/4963080e7933ee9f FAIL (2025-12-08 10:16:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448666292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................X........X......X...................X.X.] duration: 5.824368005s trace: https://jaeger.dev.storj.io/trace/76353e8535c77a45 FAIL (2025-12-08 10:16:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.824387993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.432689729s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:15:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.432723563s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................X........X......X...................X.X] duration: 1m0.000628102s trace: https://jaeger.dev.storj.io/trace/76353e8535c77a45 FAIL (2025-12-08 10:15:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000651828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.....X..............................X..........] duration: 7.300110295s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:15:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.300142697s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..................X.] duration: 5.868265482s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.868286383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..................X] duration: 2m13.186392568s trace: https://jaeger.dev.storj.io/trace/6bb63dcf18ba7880 FAIL (2025-12-08 10:15:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m16.983490049s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.112992650 (66h44m16.984075695s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113418911 (66h44m16.984501956s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.113929985 (66h44m16.98501303s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114383147 (66h44m16.985466183s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.114899351 (66h44m16.985982386s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.115393962 (66h44m16.986477007s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.116760534 (66h44m16.987843579s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.118231125 (66h44m16.98931416s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.119778110 (66h44m16.990861156s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.121927263 (66h44m16.993010308s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.124202706 (66h44m16.995285751s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.126735178 (66h44m16.997818213s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.128218984 (66h44m16.999302019s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:17.129359496 (66h44m17.000442541s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Uploading piece... Dec 8 10:13:18.574550811 (66h44m18.445633856s) [upload=58503]: Next returned nil segment Dec 8 10:13:18.574570768 (66h44m18.445653813s) [upload=58503]: Waiting for error group managing segments... Dec 8 10:13:18.961005037 (66h44m18.832088092s) [noise=true , address=5.231.73.2:4001 , piece_id=VGN3S7FOQ6PMLJU27DJMJUM464NYMN5WD2ARFI2LORFVALEIZRAA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:18.987868641 (66h44m18.858951676s) [noise=true , address=145.79.227.43:4173 , piece_id=EBJTVGC2HXJDMH24ORDVDTLJ2VVO4Y3W4RRQNRE2CMEFUCJ2MPSQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.039287472 (66h44m18.910370517s) [noise=true , address=145.79.227.24:4596 , piece_id=CMXNJSJTJNUJ7OJGEFCPUJTYAJBUCFCLXO74WMQADOZAELHNEWJQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.061246838 (66h44m18.932329883s) [noise=true , address=5.231.73.4:4036 , piece_id=IQMPMGREE7J6MOQHL2ONXVGOYWAHDTXXSSUQU75VOF3CHY23UZ7Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.098402800 (66h44m18.969485845s) [noise=true , address=77.90.31.18:4827 , piece_id=PEXOCJPLLDBRLSZ3H5MQWLMVMCXC5ZIDX3TRZRMKMY2QE7XKO4ZQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.100826505 (66h44m18.97190955s) [noise=true , address=77.90.31.19:4868 , piece_id=UOYJ6R5QQ3NYYRDXLBFLS7KLX3GNJKF4UX4DF7SY4QFS6CXMKCKQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.101134542 (66h44m18.972217587s) [noise=true , address=77.90.31.17:4800 , piece_id=XIC5XF7CCOX5RYHIOC2QG526I4POMHHYPSDXPFEQUECGREYUIGXA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.165632308 (66h44m19.036715363s) [noise=true , address=45.91.116.8:4057 , piece_id=5UOHPLONR4MPPTLIS573LNEKHGTFELGYAVUQ54IYIQPS7KG4II2A, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.168379579 (66h44m19.039462625s) [noise=true , address=77.90.31.21:4886 , piece_id=BQONW4FVXHV6JJJC3XZCODQTZ5TI7GGOBJABM76ZB4TN5CES2PQQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169085244 (66h44m19.040168289s) [noise=true , address=77.90.31.12:4598 , piece_id=HPU6ZYDGZVYKQLFGTUQFFAEC3KKB3IN2QXYKCD4LJ4JGM3EP5HYQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.169511405 (66h44m19.04059445s) [noise=true , address=45.91.116.6:4993 , piece_id=CGOM3JR6Z3MQHZY6GDYVOCPGXKOYCRQ7UDRKYSY6VTLXPDT4RJXQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.182109444 (66h44m19.053192489s) [noise=true , address=77.90.31.11:4577 , piece_id=WS7U6QAXTCBCZMYVRJJHW4QVHHAYWHRFGECQ4UE6SK7O4RBJCRAQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.197552220 (66h44m19.068635255s) [noise=true , address=45.91.116.10:4112 , piece_id=KEC6NIM6J4JNL4RWX4OXVBEDFJGEKJIIHCR6OQF5TOSVD6WAN33Q, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213459772 (66h44m19.084542817s) [noise=true , address=45.91.116.12:4248 , piece_id=TXP4E7H4KINXK3WEOH47NORAGJZMIEZBY6O3KHZWPNYFACFGQTEA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: <nil> Dec 8 10:13:19.213485701 (66h44m19.084568746s) [seg_pos={0 0}, upload=58503]: Segment reached optimal threshold of 14 pieces. Dec 8 10:13:19.213729244 (66h44m19.084812289s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34274->64.49.13.169:4710: use of closed network connection Dec 8 10:13:19.213738002 (66h44m19.084821047s) [noise=true , address=64.49.13.169:4710 , piece_id=DOIR46GL4ORAUUK3Z262XP63SSCX2KJOIWTD5PENAZCWMSCAHIBA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.213843933 (66h44m19.084926978s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo): protocol: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51932->145.79.227.38:4002: use of closed network connection Dec 8 10:13:19.213860565 (66h44m19.08494361s) [noise=true , address=145.79.227.38:4002 , piece_id=5XQ2YBQMZ3FPWRG7UFGU6PKAOSQHZ2JFRGDZQWCR7HUIS5ZZ23IQ, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214078239 (66h44m19.085161274s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214087327 (66h44m19.085170362s) [noise=true , address=145.79.227.30:4788 , piece_id=AC6TRAOAVXNGLMVQPMX3HWBHPBTN4S3XYDUADZWDK7NQUJQUCYWA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214227885 (66h44m19.08531093s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 8 10:13:19.214237402 (66h44m19.085320447s) [noise=true , address=77.90.31.16:4764 , piece_id=L6MPZIAQJQXFHMOVMCCPAT7NTEEY33EYT6CB7AUXAPGOGYMYUNPA, seg_pos={0 0}, upload=58503]: Long tail cancellation due to optimal threshold Dec 8 10:13:19.214271297 (66h44m19.085354332s) [seg_pos={0 0}, upload=58503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 10:13:19.214379413 (66h44m19.085462448s) [upload=58503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 10:13:19.214385644 (66h44m19.085468689s) [upload=58503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 10:15:27.040493313 (66h46m26.911576348s) [upload=58503]: Done uploading segments... Dec 8 10:15:27.040574868 (66h46m26.911657903s) [upload=58503]: Upload finished. err: metaclient: context deadline exceeded Duration: 2m13.186416182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.486852585s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:15:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.486884635s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.878120916s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:15:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.878152627s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.866779887s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:15:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.866801208s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.358687487s trace: https://jaeger.dev.storj.io/trace/4963080e7933ee9f FAIL (2025-12-08 10:15:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.358751449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.....X..............................X.........] duration: 5.911492685s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:14:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.911538583s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 8.522292599s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:14:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 8.522321605s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................X........X......X...................X.] duration: 5.649833687s trace: https://jaeger.dev.storj.io/trace/1101ba7f6315c4d8 FAIL (2025-12-08 10:14:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.649849076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................X........X......X...................X] duration: 1m0.001297021s trace: https://jaeger.dev.storj.io/trace/1101ba7f6315c4d8 FAIL (2025-12-08 10:14:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001315336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.418492284s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:14:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.418523665s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 10.133162621s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:14:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 10.133195344s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.877052363s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:14:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.877081169s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.434851354s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:14:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.434920948s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.435593237s trace: https://jaeger.dev.storj.io/trace/4963080e7933ee9f FAIL (2025-12-08 10:13:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435618013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.....X..............................X........] duration: 6.974412759s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:13:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.974429451s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 4.827384167s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:13:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.827423312s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 8.057989156s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:13:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 8.058017491s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.983347713s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:13:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.98337187s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.....X..............................X.......] duration: 6.050210154s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:13:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.050233347s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X........X......X...................] duration: 6.610798757s trace: https://jaeger.dev.storj.io/trace/dcdfbcaee702b97 FAIL (2025-12-08 10:13:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.610828444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.377212139s trace: https://jaeger.dev.storj.io/trace/4963080e7933ee9f FAIL (2025-12-08 10:13:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377252836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.477108981s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:13:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.477312028s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.935756796s trace: https://jaeger.dev.storj.io/trace/6ba59ddda81616b3 FAIL (2025-12-08 10:12:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.438690427 (66h36m25.309773472s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439240846 (66h36m25.310323891s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.439733043 (66h36m25.310816088s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440200392 (66h36m25.311283437s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.440655839 (66h36m25.311738884s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.441531126 (66h36m25.312614171s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442065995 (66h36m25.31314904s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.442681136 (66h36m25.313764181s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443160719 (66h36m25.314243764s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.443651053 (66h36m25.314734098s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444185170 (66h36m25.315268225s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:25.444885834 (66h36m25.31596888s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Uploading piece... Dec 8 10:05:27.267723814 (66h36m27.138806859s) [upload=58392]: Next returned nil segment Dec 8 10:05:27.267741728 (66h36m27.138824773s) [upload=58392]: Waiting for error group managing segments... Dec 8 10:05:27.571672163 (66h36m27.442755208s) [noise=true , address=77.90.31.18:4807 , piece_id=BDON2W2VREAWB7BN6D6VG4RD7JSZCEP6EVAIOUA6QW53FV4B6DFA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.589190983 (66h36m27.460274018s) [noise=true , address=77.90.31.15:4697 , piece_id=SLDAZF2PPCSVKBP2ZSOYOS7HELETNR4XFLUAAOM5PSXYY3GQTNLQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:27.757731720 (66h36m27.628814765s) [noise=true , address=77.90.31.13:4628 , piece_id=CNHLZNOWGFHMOLMGAZ7KLAL7NBTDTCX5CRR3J4B5Y6KGMHW6QIMA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:29.823436164 (66h36m29.694519199s) [noise=true , address=145.79.227.20:4437 , piece_id=5N4VSF4ISJIU5XZZ44IRHNS3EFUQUH4O6HUZJJ3FNEG7WYDA2MJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: <nil> Dec 8 10:05:31.925508293 (66h36m31.796591328s) [noise=true , address=145.79.227.44:4195 , piece_id=OGHZEZX23XMJR5UBLWQYY2W3QCLOQZFUWRKL6QWJKLL76SILZ52A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.925758539 (66h36m31.796841574s) [noise=true , address=145.79.227.40:4051 , piece_id=D6MK66OMPFNEWP3PTRR3CIVASAAE7F2RBZWFJC75H53HDDAYSGYA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.925897624 (66h36m31.796980669s) [noise=true , address=145.79.227.34:4905 , piece_id=ZNS3FRG6VANZ37W3C5LWJ7YCZ6SACEJ4ZNKMQ47FNMGP2HCMWCOA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: read tcp 188.166.225.170:60666->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926117703 (66h36m31.797200748s) [noise=true , address=145.79.227.24:4583 , piece_id=FYM7CDUHLKXGA3TM4WOVA2C6ASVNHFL5TUEZPNRSBBMD6WMD5LPA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57316->145.79.227.24:4583: use of closed network connection Dec 8 10:05:31.926182937 (66h36m31.797265972s) [noise=true , address=145.79.227.30:4762 , piece_id=Q2JMBMIQAZ6EBRXBDSATWJC4RJLXITMWRADWKFXB5KR66GMOOMJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:39216->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926364794 (66h36m31.797447839s) [noise=true , address=145.79.227.26:4653 , piece_id=4CJWY5SBICX2BACLL5VHSN7YCMS3I747BDGZ7DU56H6767OA6FJA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:44502->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926383599 (66h36m31.797466644s) [noise=true , address=145.79.227.23:4545 , piece_id=WTF2647JAOC7AE3GQDPDS5JBVWSIM5DXPK7SMXM56WYV7HQV7UCQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->145.79.227.23:4545: use of closed network connection Dec 8 10:05:31.926559444 (66h36m31.797642489s) [noise=true , address=145.79.227.23:4547 , piece_id=VE376BAJP6GZC4XORX3UYRVHKAFABLRS25PXAMMSDQQ62HHCRKHA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 10:05:31.926646370 (66h36m31.797729405s) [noise=true , address=145.79.227.43:4157 , piece_id=4IHXFIIYQQB4N5X24EZFEJQJHX524AVARRHGM7VPIWQSR6A4G73A, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:145.79.227.43:4157): protocol: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55676->145.79.227.43:4157: use of closed network connection Dec 8 10:05:31.926804901 (66h36m31.797887946s) [noise=true , address=145.79.227.38:4977 , piece_id=FWXGW2AQOWJT46A5E5YPJDL3ZHSBSASWGHYNH4QD4MBFQXLVTLFQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: read tcp 188.166.225.170:59208->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.926989644 (66h36m31.798072689s) [noise=true , address=145.79.227.22:4511 , piece_id=7AF6WXNHKB7KMIBWBMQSZ6RRIGAMMU63UISVVNVTGF4MOESRQBDQ, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:36236->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927269006 (66h36m31.798352081s) [noise=true , address=145.79.227.43:4158 , piece_id=DPK5AELFHBXMRFIJYX7DKG2WP6DX5YQPHTIXYAJBQHU6N675OHUA, seg_pos={0 0}, upload=58392]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: read tcp 188.166.225.170:57064->145.79.227.43:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 10:05:31.927317418 (66h36m31.798400463s) [seg_pos={0 0}, upload=58392]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Dec 8 10:05:32.165936106 (66h36m32.037019161s) [upload=58392]: Done uploading segments... Dec 8 10:05:32.165974188 (66h36m32.037057223s) [upload=58392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.935786242s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X........X......X..................] duration: 5.953386554s trace: https://jaeger.dev.storj.io/trace/dcdfbcaee702b97 FAIL (2025-12-08 10:12:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.953437151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.791017561s trace: https://jaeger.dev.storj.io/trace/2a7ac2b40329963b FAIL (2025-12-08 10:12:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== OBQZLQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.640147674 (66h25m5.511230709s) [noise=true , address=77.90.31.21:4910 , piece_id=E5HMU5HDLYFHXNC54JFQGI2MZJNDJLTZH4LVP64GASFZWIMNQ4BA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.640423640 (66h25m5.511506675s) [noise=true , address=45.91.116.12:4249 , piece_id=K5UHBWDKAT3SJLTCX7N6QTBAUYFUKDVWSZEAKZIY4YPKMHLF2RBQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.640741575 (66h25m5.51182462s) [noise=true , address=145.79.227.44:4218 , piece_id=XLHK65CV274RDCWYCZ2ZPUDVGNY5QJEON3SPYL7N2ZQV3HPA2ZKA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.641129513 (66h25m5.512212548s) [noise=true , address=145.79.227.24:4588 , piece_id=VX4SJQQKMVNW4WGRSQJ6SJF46SXTIVU3ZZHZF7OHI7UBYUWIOHKA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.641555204 (66h25m5.512638239s) [noise=true , address=77.90.31.15:4729 , piece_id=H6IZI4Z5Y2ZA256PSGBGCPNKPEKSOCVCJ7Z7BS752M67AV4TJ2UA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.641885273 (66h25m5.512968308s) [noise=true , address=77.90.31.13:4643 , piece_id=P6ZDVIN6ZB2II2MZUKMD7TEYMYH2HRFQIYEXBHM7CRJS7F2MVFDA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.642221823 (66h25m5.513304868s) [noise=true , address=45.91.116.14:4200 , piece_id=GXZBCEEA4V5YHJRJOV6SOFL2QGNZO5ENY5IBJS6JW7OBIHHHZK7A, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.642252872 (66h25m5.513335917s) [noise=true , address=45.91.116.8:4058 , piece_id=5ZE5OYKMBMGSHSJ54AAXRTPE2LUR2NXZF2WDMEVL5Q5NVVK7VXTA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.642508129 (66h25m5.513591164s) [noise=true , address=77.90.31.18:4833 , piece_id=A43BOZGF363Z3OTQNDAACUX2DVCBSVYT2ODGMB3PJKWEZSAC4WBA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.642762884 (66h25m5.513845929s) [noise=true , address=77.90.31.11:4557 , piece_id=6J3IEWOVWZMVTGOW6HXZSPHDG5V2WLGADZJQUHWYMI5B7DDCWFSQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.643403463 (66h25m5.514486498s) [noise=true , address=45.91.116.11:4135 , piece_id=4WI3JKN7A7JRU6NMPQGGQFK2Z7ZK4CH4NZH7WUMTOAIW6A4PVTGQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.643854762 (66h25m5.514937797s) [noise=true , address=77.90.31.10:4523 , piece_id=EJ54WQYV6YMKBP4NWTQB42WJ4HXKLNT4A6LPNMBX35G4SX6GO3NA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.644341670 (66h25m5.515424705s) [noise=true , address=45.91.116.6:4004 , piece_id=OC7IJ4FTMHDLK65UIVXEQKC7VYLZ6PRS6RV5FVW7N6TQLV2VOVJA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.644962332 (66h25m5.516045377s) [noise=true , address=77.90.31.19:4858 , piece_id=7XJYLRMZ5YIETVIBCHBHFIMTSQG45QHINWJJDW3TX6D66QJ65ODA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.645270268 (66h25m5.516353303s) [noise=true , address=45.91.116.7:4041 , piece_id=J646AZOI4CIVFZ247KATPMA562FPYI6FCXRNJITTCA2HSS3ZQ26A, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.645789447 (66h25m5.516872492s) [noise=true , address=145.79.227.30:4796 , piece_id=CLMMQ2FCPXXRQUHJY5WVA4ITWY6MF4EF42TGAZI5VU4WQX5HWSBQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:07.172418522 (66h25m7.043501567s) [upload=58247]: Next returned nil segment Dec 8 09:54:07.172438419 (66h25m7.043521464s) [upload=58247]: Waiting for error group managing segments... Dec 8 09:54:07.606175924 (66h25m7.477258959s) [noise=true , address=145.79.227.24:4588 , piece_id=VX4SJQQKMVNW4WGRSQJ6SJF46SXTIVU3ZZHZF7OHI7UBYUWIOHKA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.622732341 (66h25m7.493815376s) [noise=true , address=145.79.227.44:4218 , piece_id=XLHK65CV274RDCWYCZ2ZPUDVGNY5QJEON3SPYL7N2ZQV3HPA2ZKA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.633241162 (66h25m7.504324207s) [noise=true , address=145.79.227.30:4796 , piece_id=CLMMQ2FCPXXRQUHJY5WVA4ITWY6MF4EF42TGAZI5VU4WQX5HWSBQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.638428720 (66h25m7.509511765s) [noise=true , address=77.90.31.11:4557 , piece_id=6J3IEWOVWZMVTGOW6HXZSPHDG5V2WLGADZJQUHWYMI5B7DDCWFSQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.650560211 (66h25m7.521643256s) [noise=true , address=77.90.31.12:4602 , piece_id=CKWCFGDLBJJMWO77RZYQFHEOCJEN3BMPI7SAWZLQ7OO7POOBQZLQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.658671337 (66h25m7.529754382s) [noise=true , address=77.90.31.10:4523 , piece_id=EJ54WQYV6YMKBP4NWTQB42WJ4HXKLNT4A6LPNMBX35G4SX6GO3NA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.669910919 (66h25m7.540993954s) [noise=true , address=77.90.31.15:4729 , piece_id=H6IZI4Z5Y2ZA256PSGBGCPNKPEKSOCVCJ7Z7BS752M67AV4TJ2UA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.680617957 (66h25m7.551701003s) [noise=true , address=77.90.31.18:4833 , piece_id=A43BOZGF363Z3OTQNDAACUX2DVCBSVYT2ODGMB3PJKWEZSAC4WBA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.693123821 (66h25m7.564206866s) [noise=true , address=77.90.31.13:4643 , piece_id=P6ZDVIN6ZB2II2MZUKMD7TEYMYH2HRFQIYEXBHM7CRJS7F2MVFDA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.694735520 (66h25m7.565818565s) [noise=true , address=45.91.116.10:4084 , piece_id=PBZFVHTTBSDGFRNGD7QL2IM25PAYDW3AOXUA7YZTZXZJ7BXF6CYQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.695359188 (66h25m7.566442233s) [noise=true , address=45.91.116.6:4004 , piece_id=OC7IJ4FTMHDLK65UIVXEQKC7VYLZ6PRS6RV5FVW7N6TQLV2VOVJA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.709334970 (66h25m7.580418015s) [noise=true , address=45.91.116.12:4249 , piece_id=K5UHBWDKAT3SJLTCX7N6QTBAUYFUKDVWSZEAKZIY4YPKMHLF2RBQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.733188052 (66h25m7.604271097s) [noise=true , address=45.91.116.8:4058 , piece_id=5ZE5OYKMBMGSHSJ54AAXRTPE2LUR2NXZF2WDMEVL5Q5NVVK7VXTA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.733281850 (66h25m7.604364905s) [noise=true , address=45.91.116.14:4200 , piece_id=GXZBCEEA4V5YHJRJOV6SOFL2QGNZO5ENY5IBJS6JW7OBIHHHZK7A, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.733296468 (66h25m7.604379513s) [seg_pos={0 0}, upload=58247]: Segment reached optimal threshold of 14 pieces. Dec 8 09:54:07.733648338 (66h25m7.604731393s) [noise=true , address=77.90.31.19:4858 , piece_id=7XJYLRMZ5YIETVIBCHBHFIMTSQG45QHINWJJDW3TX6D66QJ65ODA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 8 09:54:07.733664408 (66h25m7.604747473s) [noise=true , address=77.90.31.19:4858 , piece_id=7XJYLRMZ5YIETVIBCHBHFIMTSQG45QHINWJJDW3TX6D66QJ65ODA, seg_pos={0 0}, upload=58247]: Long tail cancellation due to optimal threshold Dec 8 09:54:07.734072546 (66h25m7.605155591s) [noise=true , address=77.90.31.21:4910 , piece_id=E5HMU5HDLYFHXNC54JFQGI2MZJNDJLTZH4LVP64GASFZWIMNQ4BA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H): protocol: write tcp 188.166.225.170:36900->77.90.31.21:4910: use of closed network connection; write tcp 188.166.225.170:36900->77.90.31.21:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36900->77.90.31.21:4910: use of closed network connection Dec 8 09:54:07.734092053 (66h25m7.605175098s) [noise=true , address=77.90.31.21:4910 , piece_id=E5HMU5HDLYFHXNC54JFQGI2MZJNDJLTZH4LVP64GASFZWIMNQ4BA, seg_pos={0 0}, upload=58247]: Long tail cancellation due to optimal threshold Dec 8 09:54:07.734124665 (66h25m7.60520771s) [noise=true , address=45.91.116.11:4135 , piece_id=4WI3JKN7A7JRU6NMPQGGQFK2Z7ZK4CH4NZH7WUMTOAIW6A4PVTGQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:194doyGN13Ha8hsyjrERgTZqdyzFhdTQyiTGQg9rFA7CxvKmkM): protocol: expected piece hash; context canceled; EOF Dec 8 09:54:07.734135175 (66h25m7.60521822s) [noise=true , address=45.91.116.11:4135 , piece_id=4WI3JKN7A7JRU6NMPQGGQFK2Z7ZK4CH4NZH7WUMTOAIW6A4PVTGQ, seg_pos={0 0}, upload=58247]: Long tail cancellation due to optimal threshold Dec 8 09:54:07.734343943 (66h25m7.605426988s) [noise=true , address=45.91.116.7:4041 , piece_id=J646AZOI4CIVFZ247KATPMA562FPYI6FCXRNJITTCA2HSS3ZQ26A, seg_pos={0 0}, upload=58247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV): protocol: expected piece hash; context canceled; EOF Dec 8 09:54:07.734354653 (66h25m7.605437698s) [noise=true , address=45.91.116.7:4041 , piece_id=J646AZOI4CIVFZ247KATPMA562FPYI6FCXRNJITTCA2HSS3ZQ26A, seg_pos={0 0}, upload=58247]: Long tail cancellation due to optimal threshold Dec 8 09:54:07.734405350 (66h25m7.605488395s) [seg_pos={0 0}, upload=58247]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 09:54:07.734496523 (66h25m7.605579568s) [upload=58247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 09:54:07.734503396 (66h25m7.605586441s) [upload=58247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 09:59:46.381411728 (66h30m46.252494773s) [upload=58247]: Done uploading segments... Dec 8 09:59:46.381474758 (66h30m46.252557804s) [upload=58247]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.791180763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.....X..............................X......] duration: 7.247349126s trace: https://jaeger.dev.storj.io/trace/4939b1d48dfb8f81 FAIL (2025-12-08 10:12:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.247383381s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.088417235s trace: https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3 FAIL (2025-12-08 10:12:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.088448996s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 7.127797023s trace: https://jaeger.dev.storj.io/trace/2a7ac2b40329963b FAIL (2025-12-08 10:12:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== OBQZLQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.640147674 (66h25m5.511230709s) [noise=true , address=77.90.31.21:4910 , piece_id=E5HMU5HDLYFHXNC54JFQGI2MZJNDJLTZH4LVP64GASFZWIMNQ4BA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.640423640 (66h25m5.511506675s) [noise=true , address=45.91.116.12:4249 , piece_id=K5UHBWDKAT3SJLTCX7N6QTBAUYFUKDVWSZEAKZIY4YPKMHLF2RBQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.640741575 (66h25m5.51182462s) [noise=true , address=145.79.227.44:4218 , piece_id=XLHK65CV274RDCWYCZ2ZPUDVGNY5QJEON3SPYL7N2ZQV3HPA2ZKA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.641129513 (66h25m5.512212548s) [noise=true , address=145.79.227.24:4588 , piece_id=VX4SJQQKMVNW4WGRSQJ6SJF46SXTIVU3ZZHZF7OHI7UBYUWIOHKA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.641555204 (66h25m5.512638239s) [noise=true , address=77.90.31.15:4729 , piece_id=H6IZI4Z5Y2ZA256PSGBGCPNKPEKSOCVCJ7Z7BS752M67AV4TJ2UA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.641885273 (66h25m5.512968308s) [noise=true , address=77.90.31.13:4643 , piece_id=P6ZDVIN6ZB2II2MZUKMD7TEYMYH2HRFQIYEXBHM7CRJS7F2MVFDA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.642221823 (66h25m5.513304868s) [noise=true , address=45.91.116.14:4200 , piece_id=GXZBCEEA4V5YHJRJOV6SOFL2QGNZO5ENY5IBJS6JW7OBIHHHZK7A, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.642252872 (66h25m5.513335917s) [noise=true , address=45.91.116.8:4058 , piece_id=5ZE5OYKMBMGSHSJ54AAXRTPE2LUR2NXZF2WDMEVL5Q5NVVK7VXTA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.642508129 (66h25m5.513591164s) [noise=true , address=77.90.31.18:4833 , piece_id=A43BOZGF363Z3OTQNDAACUX2DVCBSVYT2ODGMB3PJKWEZSAC4WBA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.642762884 (66h25m5.513845929s) [noise=true , address=77.90.31.11:4557 , piece_id=6J3IEWOVWZMVTGOW6HXZSPHDG5V2WLGADZJQUHWYMI5B7DDCWFSQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.643403463 (66h25m5.514486498s) [noise=true , address=45.91.116.11:4135 , piece_id=4WI3JKN7A7JRU6NMPQGGQFK2Z7ZK4CH4NZH7WUMTOAIW6A4PVTGQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.643854762 (66h25m5.514937797s) [noise=true , address=77.90.31.10:4523 , piece_id=EJ54WQYV6YMKBP4NWTQB42WJ4HXKLNT4A6LPNMBX35G4SX6GO3NA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.644341670 (66h25m5.515424705s) [noise=true , address=45.91.116.6:4004 , piece_id=OC7IJ4FTMHDLK65UIVXEQKC7VYLZ6PRS6RV5FVW7N6TQLV2VOVJA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.644962332 (66h25m5.516045377s) [noise=true , address=77.90.31.19:4858 , piece_id=7XJYLRMZ5YIETVIBCHBHFIMTSQG45QHINWJJDW3TX6D66QJ65ODA, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.645270268 (66h25m5.516353303s) [noise=true , address=45.91.116.7:4041 , piece_id=J646AZOI4CIVFZ247KATPMA562FPYI6FCXRNJITTCA2HSS3ZQ26A, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:05.645789447 (66h25m5.516872492s) [noise=true , address=145.79.227.30:4796 , piece_id=CLMMQ2FCPXXRQUHJY5WVA4ITWY6MF4EF42TGAZI5VU4WQX5HWSBQ, seg_pos={0 0}, upload=58247]: Uploading piece... Dec 8 09:54:07.172418522 (66h25m7.043501567s) [upload=58247]: Next returned nil segment Dec 8 09:54:07.172438419 (66h25m7.043521464s) [upload=58247]: Waiting for error group managing segments... Dec 8 09:54:07.606175924 (66h25m7.477258959s) [noise=true , address=145.79.227.24:4588 , piece_id=VX4SJQQKMVNW4WGRSQJ6SJF46SXTIVU3ZZHZF7OHI7UBYUWIOHKA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.622732341 (66h25m7.493815376s) [noise=true , address=145.79.227.44:4218 , piece_id=XLHK65CV274RDCWYCZ2ZPUDVGNY5QJEON3SPYL7N2ZQV3HPA2ZKA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.633241162 (66h25m7.504324207s) [noise=true , address=145.79.227.30:4796 , piece_id=CLMMQ2FCPXXRQUHJY5WVA4ITWY6MF4EF42TGAZI5VU4WQX5HWSBQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.638428720 (66h25m7.509511765s) [noise=true , address=77.90.31.11:4557 , piece_id=6J3IEWOVWZMVTGOW6HXZSPHDG5V2WLGADZJQUHWYMI5B7DDCWFSQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.650560211 (66h25m7.521643256s) [noise=true , address=77.90.31.12:4602 , piece_id=CKWCFGDLBJJMWO77RZYQFHEOCJEN3BMPI7SAWZLQ7OO7POOBQZLQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.658671337 (66h25m7.529754382s) [noise=true , address=77.90.31.10:4523 , piece_id=EJ54WQYV6YMKBP4NWTQB42WJ4HXKLNT4A6LPNMBX35G4SX6GO3NA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.669910919 (66h25m7.540993954s) [noise=true , address=77.90.31.15:4729 , piece_id=H6IZI4Z5Y2ZA256PSGBGCPNKPEKSOCVCJ7Z7BS752M67AV4TJ2UA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.680617957 (66h25m7.551701003s) [noise=true , address=77.90.31.18:4833 , piece_id=A43BOZGF363Z3OTQNDAACUX2DVCBSVYT2ODGMB3PJKWEZSAC4WBA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.693123821 (66h25m7.564206866s) [noise=true , address=77.90.31.13:4643 , piece_id=P6ZDVIN6ZB2II2MZUKMD7TEYMYH2HRFQIYEXBHM7CRJS7F2MVFDA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.694735520 (66h25m7.565818565s) [noise=true , address=45.91.116.10:4084 , piece_id=PBZFVHTTBSDGFRNGD7QL2IM25PAYDW3AOXUA7YZTZXZJ7BXF6CYQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.695359188 (66h25m7.566442233s) [noise=true , address=45.91.116.6:4004 , piece_id=OC7IJ4FTMHDLK65UIVXEQKC7VYLZ6PRS6RV5FVW7N6TQLV2VOVJA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.709334970 (66h25m7.580418015s) [noise=true , address=45.91.116.12:4249 , piece_id=K5UHBWDKAT3SJLTCX7N6QTBAUYFUKDVWSZEAKZIY4YPKMHLF2RBQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.733188052 (66h25m7.604271097s) [noise=true , address=45.91.116.8:4058 , piece_id=5ZE5OYKMBMGSHSJ54AAXRTPE2LUR2NXZF2WDMEVL5Q5NVVK7VXTA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.733281850 (66h25m7.604364905s) [noise=true , address=45.91.116.14:4200 , piece_id=GXZBCEEA4V5YHJRJOV6SOFL2QGNZO5ENY5IBJS6JW7OBIHHHZK7A, seg_pos={0 0}, upload=58247]: Done uploading piece. err: <nil> Dec 8 09:54:07.733296468 (66h25m7.604379513s) [seg_pos={0 0}, upload=58247]: Segment reached optimal threshold of 14 pieces. Dec 8 09:54:07.733648338 (66h25m7.604731393s) [noise=true , address=77.90.31.19:4858 , piece_id=7XJYLRMZ5YIETVIBCHBHFIMTSQG45QHINWJJDW3TX6D66QJ65ODA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 8 09:54:07.733664408 (66h25m7.604747473s) [noise=true , address=77.90.31.19:4858 , piece_id=7XJYLRMZ5YIETVIBCHBHFIMTSQG45QHINWJJDW3TX6D66QJ65ODA, seg_pos={0 0}, upload=58247]: Long tail cancellation due to optimal threshold Dec 8 09:54:07.734072546 (66h25m7.605155591s) [noise=true , address=77.90.31.21:4910 , piece_id=E5HMU5HDLYFHXNC54JFQGI2MZJNDJLTZH4LVP64GASFZWIMNQ4BA, seg_pos={0 0}, upload=58247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H): protocol: write tcp 188.166.225.170:36900->77.90.31.21:4910: use of closed network connection; write tcp 188.166.225.170:36900->77.90.31.21:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36900->77.90.31.21:4910: use of closed network connection Dec 8 09:54:07.734092053 (66h25m7.605175098s) [noise=true , address=77.90.31.21:4910 , piece_id=E5HMU5HDLYFHXNC54JFQGI2MZJNDJLTZH4LVP64GASFZWIMNQ4BA, seg_pos={0 0}, upload=58247]: Long tail cancellation due to optimal threshold Dec 8 09:54:07.734124665 (66h25m7.60520771s) [noise=true , address=45.91.116.11:4135 , piece_id=4WI3JKN7A7JRU6NMPQGGQFK2Z7ZK4CH4NZH7WUMTOAIW6A4PVTGQ, seg_pos={0 0}, upload=58247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:194doyGN13Ha8hsyjrERgTZqdyzFhdTQyiTGQg9rFA7CxvKmkM): protocol: expected piece hash; context canceled; EOF Dec 8 09:54:07.734135175 (66h25m7.60521822s) [noise=true , address=45.91.116.11:4135 , piece_id=4WI3JKN7A7JRU6NMPQGGQFK2Z7ZK4CH4NZH7WUMTOAIW6A4PVTGQ, seg_pos={0 0}, upload=58247]: Long tail cancellation due to optimal threshold Dec 8 09:54:07.734343943 (66h25m7.605426988s) [noise=true , address=45.91.116.7:4041 , piece_id=J646AZOI4CIVFZ247KATPMA562FPYI6FCXRNJITTCA2HSS3ZQ26A, seg_pos={0 0}, upload=58247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV): protocol: expected piece hash; context canceled; EOF Dec 8 09:54:07.734354653 (66h25m7.605437698s) [noise=true , address=45.91.116.7:4041 , piece_id=J646AZOI4CIVFZ247KATPMA562FPYI6FCXRNJITTCA2HSS3ZQ26A, seg_pos={0 0}, upload=58247]: Long tail cancellation due to optimal threshold Dec 8 09:54:07.734405350 (66h25m7.605488395s) [seg_pos={0 0}, upload=58247]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 09:54:07.734496523 (66h25m7.605579568s) [upload=58247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 09:54:07.734503396 (66h25m7.605586441s) [upload=58247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 09:59:46.381411728 (66h30m46.252494773s) [upload=58247]: Done uploading segments... Dec 8 09:59:46.381474758 (66h30m46.252557804s) [upload=58247]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.127843641s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.599066915s trace: https://jaeger.dev.storj.io/trace/4963080e7933ee9f FAIL (2025-12-08 10:12:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.599096932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.401349632s trace: https://jaeger.dev.storj.io/trace/3d2642fe33f63575 FAIL (2025-12-08 10:11:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.401370791s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)