Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.552122396s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:41:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.552147142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................X................................] duration: 5.835190874s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:41:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.835227995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.681411164s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:40:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.68143561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.................X...............................] duration: 5.953260955s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:40:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.953287185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.................X..............................] duration: 5.806203744s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:39:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.806230395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.544104247s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:39:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.544132251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.................X.............................] duration: 5.794495006s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:39:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.794525333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.495099008s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:39:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 6.495129547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.................X............................] duration: 5.725780649s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:38:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.725810134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.37818946s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:38:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.378219778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.................X...........................] duration: 5.671073039s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:38:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.67111546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.581641841s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:37:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.5816718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.................X..........................] duration: 5.762212799s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:37:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.762240621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.657283232s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:36:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.6573078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.................X.........................] duration: 5.795804973s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:36:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.795831844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.744767032s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:35:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.744794183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.................X........................] duration: 5.945170481s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:35:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.945188845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.................X.......................] duration: 5.7857615s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:35:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.785789043s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.74240407s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:35:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.742435549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.52384459s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:34:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.523871361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.................X......................] duration: 5.852256754s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.852287843s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.626981322s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:33:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.627025165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................X.....................] duration: 5.746104069s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:33:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.746138324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................X....................] duration: 5.788888172s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:32:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.788918991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.498100333s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:32:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.498126473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.691258038s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:32:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.691288375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................X...................] duration: 5.689490227s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:32:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.68952876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.638593664s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:31:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.638623723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.................X..................] duration: 5.805441092s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:31:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.805469775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.560875082s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:31:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.560912943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.................X.................] duration: 5.817783084s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.817810907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.374430765s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:30:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 6.374462283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.................X................] duration: 5.754912719s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:29:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.754939441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.482404735s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:29:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.482432015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.517260556s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.517282287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.................X...............] duration: 7.388538744s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:28:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 7.388565465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.561330658s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:28:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.561360677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.................X..............] duration: 5.76369523s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:27:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.763723363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.589326592s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:27:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.589353402s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.................X.............] duration: 5.867723711s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:26:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.867755562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.70292597s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:26:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.702954215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.................X............] duration: 5.747399806s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:26:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.747425575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.539582647s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:25:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.539612554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.817740047s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:25:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.817763151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.................X...........] duration: 6.168246619s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:25:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 6.168266717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.724096231s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:25:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.724140826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.489793061s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:24:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.489825944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.................X..........] duration: 5.705960319s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:24:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.70598708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.................X.........] duration: 5.763922745s trace: https://jaeger.dev.storj.io/trace/58d1ffc666c9fb15 FAIL (2025-12-26 02:23:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 02:16:35.923594404 (490h47m35.794677439s) [upload=430749]: Starting upload Dec 26 02:16:35.923678334 (490h47m35.794761379s) [upload=430749]: Done starting upload Dec 26 02:16:35.923786129 (490h47m35.794869174s) [upload=430749]: Uploading segments... Dec 26 02:16:35.923935734 (490h47m35.795018769s) [upload=430749]: Got next segment. Inline: false Dec 26 02:16:35.923967695 (490h47m35.79505073s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 02:16:35.923972844 (490h47m35.79505589s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 02:16:36.171304298 (490h47m36.042387333s) [seg_pos={0 0}, upload=430749]: Begin upload segment... Dec 26 02:16:36.171370605 (490h47m36.04245365s) [seg_pos={0 0}, upload=430749]: Done begin upload segment. Dec 26 02:16:36.171381996 (490h47m36.042465031s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.171407906 (490h47m36.042490951s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172267752 (490h47m36.043350797s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172567964 (490h47m36.043651009s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.172826115 (490h47m36.04390915s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173085340 (490h47m36.044168635s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173350083 (490h47m36.044433118s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173611722 (490h47m36.044694757s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.173949976 (490h47m36.045033011s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174395423 (490h47m36.045478459s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.174868254 (490h47m36.045951299s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.175909998 (490h47m36.046993033s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.176081194 (490h47m36.047164229s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.177642536 (490h47m36.048725581s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.179839530 (490h47m36.050922565s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:36.181382899 (490h47m36.052465934s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Uploading piece... Dec 26 02:16:38.030541482 (490h47m37.901624517s) [upload=430749]: Next returned nil segment Dec 26 02:16:38.030558054 (490h47m37.901641089s) [upload=430749]: Waiting for error group managing segments... Dec 26 02:16:38.341774904 (490h47m38.212857939s) [noise=true , address=145.79.227.46:4603 , piece_id=7ZRECWD4LFWL5YN6LPMIQWOE2W3T5DT3A4UCPBIMNOCNJOLVFZQQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.348154832 (490h47m38.219237867s) [noise=true , address=45.91.116.5:4300 , piece_id=KQIOFR47TTXOXNA6Q3XVVGFDTMBO3CO4RSDXIQPNF7EJGZGNCNGQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.349003679 (490h47m38.220086714s) [noise=true , address=145.79.227.45:4231 , piece_id=JVVR25GSKKTY5JDQAYYLSPFVY32QNM3TY2OE2UVEMNKCYTGNSUHA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.353359443 (490h47m38.224442478s) [noise=true , address=45.91.116.14:4190 , piece_id=MIW3TMBJPPMV45533UHVOLSDDVFYIXSVGILTQNMDUA3DVNYPLC5Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.384219895 (490h47m38.25530294s) [noise=true , address=145.79.227.50:4373 , piece_id=XICV3WS2RO3D4QU5R6CUTRI4K4A3G6AG27LANUWKEH5W7ALYR46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.390734690 (490h47m38.261817725s) [noise=true , address=145.79.227.41:4085 , piece_id=DS5TNKNNDIALTODZKWWJOQQOUYJ3CRMADQETNXJP6GJW5YRXFL2A, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:38.391181381 (490h47m38.262264416s) [noise=true , address=145.79.227.47:4267 , piece_id=R4WPIBBZRAAAU6YROY2QK636V35CNYILTKIKBUAPR23ZOFCSRARQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.036835907 (490h47m38.907918942s) [noise=true , address=145.79.227.29:4942 , piece_id=VK2U7WHKSXJ6JVPXPLEA5ZDSDJH64NSQBM6B6QPWCHQ2B4CPCCQA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.135704507 (490h47m39.006787542s) [noise=true , address=45.91.116.12:4228 , piece_id=NSINPV5EM5BGCFTSJYY7AK4WUNOBBFXTF7CC2HHNUVIUGVXKHSUQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.141432554 (490h47m39.012515589s) [noise=true , address=45.91.116.4:4939 , piece_id=56ND2XTSVQ7WFMP7BFLJU35V32E2GFZM3QK6NLJJGO6J75KJPWPA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.365662639 (490h47m39.236745674s) [noise=true , address=45.91.116.7:4011 , piece_id=DUXIIR3BBKGVFAOJDEYKGOUWC6TCU4OE7L3LNP3MVS7X52W3T5NA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.498687767 (490h47m39.369770802s) [noise=true , address=145.79.227.14:4839 , piece_id=BUKNSENFT5DIKIGNCLPZHXJIKVTPFTWCLOAWAQKDGCPJUAL5B46Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759811451 (490h47m39.630894486s) [noise=true , address=145.79.227.44:4195 , piece_id=ZUSNNTHOI337GGBYBCVWSDGCFGDU7ZMXPXXGZG5NIRBSV3VF2DLQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: <nil> Dec 26 02:16:39.759834516 (490h47m39.630917551s) [seg_pos={0 0}, upload=430749]: Segment reached optimal threshold of 13 pieces. Dec 26 02:16:39.760131442 (490h47m39.631214477s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760140709 (490h47m39.631223754s) [noise=true , address=145.79.227.34:4906 , piece_id=3O22MLVYFW4ZJPRY72TVRY53DWSTJIKBTJSBW22JEGM7LI4DZ5YQ, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760266007 (490h47m39.631349052s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:39.760287037 (490h47m39.631370092s) [noise=true , address=145.79.227.11:4732 , piece_id=SN56HVMEYREJN6EQ434C4PSH2OIGG6HFXNE3STB2LEK3OKYDPFLA, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760456280 (490h47m39.631539325s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:56820->145.79.227.7:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:39.760466148 (490h47m39.631549193s) [noise=true , address=145.79.227.7:4294 , piece_id=BUGPUO5C6EKAWTJ446ILVIOWZTUUXRKXSXGT7SEPNLDRFTU2UK7Q, seg_pos={0 0}, upload=430749]: Long tail cancellation due to optimal threshold Dec 26 02:16:39.760488972 (490h47m39.631572007s) [seg_pos={0 0}, upload=430749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 02:16:39.760573292 (490h47m39.631656337s) [upload=430749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:39.760579143 (490h47m39.631662188s) [upload=430749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:40.056409957 (490h47m39.927493012s) [upload=430749]: Done uploading segments... Dec 26 02:16:40.056433532 (490h47m39.927516567s) [upload=430749]: Upload finished. err: <nil> Duration: 5.763954856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.625716423s trace: https://jaeger.dev.storj.io/trace/6ed5083a0488af0e FAIL (2025-12-26 02:23:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, Piece ID: MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA) context deadline exceeded; piecestore: (Node ID: 1qxNK1SVXKBZJnNpf3iyhtqrBXD7KcRwJDpru1uz8DZQ45gNTJ, Piece ID: ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ) context deadline exceeded; piecestore: (Node ID: 12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, Piece ID: BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A) context deadline exceeded; piecestore: (Node ID: 1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, Piece ID: 7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A) context deadline exceeded; piecestore: (Node ID: 12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, Piece ID: SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA) context deadline exceeded; piecestore: (Node ID: 1uD5YHqzjrr7kVJTpca8KCQUCmRkFwN5hD1ZnCLvDhnSiaofrw, Piece ID: N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ) context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.864876311 (490h47m14.735959346s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865239922 (490h47m14.736322958s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865535065 (490h47m14.73661811s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865890582 (490h47m14.736973617s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.865965534 (490h47m14.737048599s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866501605 (490h47m14.73758465s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.866819801 (490h47m14.737902847s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867098602 (490h47m14.738181637s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867467604 (490h47m14.738550639s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.867834433 (490h47m14.738917478s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868201502 (490h47m14.739284537s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.868537752 (490h47m14.739620787s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870461515 (490h47m14.74154456s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.870892506 (490h47m14.741975551s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:14.871404351 (490h47m14.742487386s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Uploading piece... Dec 26 02:16:16.166020611 (490h47m16.037103656s) [upload=430745]: Next returned nil segment Dec 26 02:16:16.166037152 (490h47m16.037120197s) [upload=430745]: Waiting for error group managing segments... Dec 26 02:16:16.508383178 (490h47m16.379466213s) [noise=true , address=64.49.13.139:4471 , piece_id=AI6UYQWD5CW5URG6O3KDNLF6OD2PNZIGJ3MW32U3K5R32LDPZJLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.891230532 (490h47m16.762313577s) [noise=true , address=145.79.227.13:4836 , piece_id=DGSAVIJIPZWAQAIOO3AZRGABTSKRS26N4AMMXGEHEMCPO65U2J5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.904142879 (490h47m16.775225914s) [noise=true , address=77.90.31.16:4744 , piece_id=BJCWQ53U6K3IKFR77MYLPNCNQ3TPDTV4KZVWQALZVEKTG6C2FW5A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.941732003 (490h47m16.812815038s) [noise=true , address=77.90.31.18:4836 , piece_id=XDY6234OXE2WP2OPKTEICH7QGTNLY4CPUZCTEED73RHTJVKLSZSA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.946736653 (490h47m16.817819698s) [noise=true , address=145.79.227.38:4003 , piece_id=7GDIF7YGEPL7WFTMRC4CQLM5JMBWXFQ4BZEP6QIFXKCWZLZYCZ2A, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.952660773 (490h47m16.823743819s) [noise=true , address=145.79.227.23:4557 , piece_id=SPC4F7U7425QLDEXUJIESJQFF3YOGCWDZDL2Y3FILYVQG7GACWXA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:16.956014429 (490h47m16.827097474s) [noise=true , address=45.91.116.5:4317 , piece_id=N2E2L4HWB5RLLD3ROQD7DQ4MXDPSOG3QLILMLCLARS3EBEQQCQTQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.644784984 (490h47m17.515868029s) [noise=true , address=77.90.31.13:4646 , piece_id=HV33YLQK2HMKD664CBSLZK2KDSCQ7HWI4KXRK4RFAHRAJQ3F6UJQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.666595785 (490h47m17.53767883s) [noise=true , address=145.79.227.22:4544 , piece_id=C4SJG3GHHERRIT7IQDXON736Q2VLL4ZFTLIFG4LQUVCF5JOT3TBQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.676573724 (490h47m17.547656779s) [noise=true , address=145.79.227.32:4858 , piece_id=QINIPXM3YHWDUYLMLIJT4DKOVUR2U5LWS26O3SCTG4MGYS5SCGVA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.838561541 (490h47m17.709644576s) [noise=true , address=145.79.227.30:4782 , piece_id=OEUC6AMDQNMAC5SBBKBUHWOIYVX6J7FMSP6DCKNYUVXXTJADIAYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:17.933927551 (490h47m17.805010586s) [noise=true , address=145.79.227.33:4881 , piece_id=MPYYM7Q2B5ZWIOHYM7QLZY5RFGT6HZNEDOZAOCIT4YVO2RYZ7QLA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.333037029 (490h47m18.204120065s) [noise=true , address=77.90.31.15:4705 , piece_id=Q2MSMZJT5XBTWWOH6H4GZMNKSZRKJLM4BJEAUHF2Y6QEPAOQJFYA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459612658 (490h47m18.330695703s) [noise=true , address=45.91.116.13:4187 , piece_id=ROYMYN4MFWXK2NUTTVGAB4AHWAXWYKF3MVBED4S7EBVYXHH2K3WQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: <nil> Dec 26 02:16:18.459635060 (490h47m18.330718105s) [seg_pos={0 0}, upload=430745]: Segment reached optimal threshold of 14 pieces. Dec 26 02:16:18.459816727 (490h47m18.330899762s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.459823129 (490h47m18.330906174s) [noise=true , address=145.79.227.40:4078 , piece_id=UBX3VXNDSX3CWJ6R75PIAW7HKWEZ5EJXNRA35Q4F34M5JEJT3CDA, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460072203 (490h47m18.331155238s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 26 02:16:18.460078475 (490h47m18.33116152s) [noise=true , address=77.90.31.21:4907 , piece_id=EVARQIV4TDJDVVSEEIA2USZUXQ6PJV3DQRNUFGUYI6O7UKGSC57Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460267695 (490h47m18.33135074s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW): protocol: read tcp 188.166.225.170:48918->145.79.227.18:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 02:16:18.460273426 (490h47m18.331356472s) [noise=true , address=145.79.227.18:4013 , piece_id=NXPOQN5FXALKVZNN73AKTIT5HUDP242JZ2N7CANQCPR2WXKKEDNQ, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460358598 (490h47m18.331441633s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34974->64.49.13.169:4706: use of closed network connection Dec 26 02:16:18.460364409 (490h47m18.331447444s) [noise=true , address=64.49.13.169:4706 , piece_id=DZVHZXACM423FC4XF6O6VGE5KYG7LBXGE5EBPMXU5OXPLVSR5M6Q, seg_pos={0 0}, upload=430745]: Long tail cancellation due to optimal threshold Dec 26 02:16:18.460378426 (490h47m18.331461461s) [seg_pos={0 0}, upload=430745]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 02:16:18.460450484 (490h47m18.331533529s) [upload=430745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 02:16:18.460454852 (490h47m18.331537897s) [upload=430745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 02:16:18.742769578 (490h47m18.613852613s) [upload=430745]: Done uploading segments... Dec 26 02:16:18.742789216 (490h47m18.613872261s) [upload=430745]: Upload finished. err: <nil> Duration: 5.625748804s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)