Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration:
5.00204534s trace:
https://jaeger.dev.storj.io/trace/5f25bb9425be5307
FAIL (2026-01-09 11:08:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002067914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.XX.......X.......X.X...X...XXX......X..........X.........XX] duration:
30.000833743s trace:
https://jaeger.dev.storj.io/trace/47aceef065d7557
FAIL (2026-01-09 11:08:19 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000854443s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.5928018s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:08:06 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.592829392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.878802716s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:07:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.878830959s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..XX.......X.......X.X...X...XXX......X..........X.........X] duration:
30.001096905s trace:
https://jaeger.dev.storj.io/trace/48059283609a223
FAIL (2026-01-09 11:07:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00111041s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.499092858s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:07:44 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.499116925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
5.001218268s trace:
https://jaeger.dev.storj.io/trace/73dda180d0349a20
FAIL (2026-01-09 11:07:42 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001247245s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.778247154s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:06:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.778276059s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.510053186s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:06:42 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.510082912s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.002085779s trace:
https://jaeger.dev.storj.io/trace/73dda180d0349a20
FAIL (2026-01-09 11:06:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002109544s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...XX.......X.......X.X...X...XXX......X..........X.........] duration:
25.042638943s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:06:36 UTC)
orbiter warning: context deadline exceeded
Duration: 25.042664472s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
5.506569769s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:06:04 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.506595848s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.743482956s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:05:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.743500368s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000595099s trace:
https://jaeger.dev.storj.io/trace/69756dab51a10bc7
FAIL (2026-01-09 11:05:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000617452s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [....XX.......X.......X.X...X...XXX......X..........X........] duration:
19.130595004s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:05:32 UTC)
orbiter warning: context deadline exceeded
Duration: 19.130626563s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.....XX.......X.......X.X...X...XXX......X..........X.......] duration:
28.9045897s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:05:12 UTC)
orbiter warning: context deadline exceeded
Duration: 28.904613424s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
5.652669918s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:05:04 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.652701066s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
5.775498452s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:04:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.775522277s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001345858s trace:
https://jaeger.dev.storj.io/trace/7213ec5c9c200eee
FAIL (2026-01-09 11:04:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001367972s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [......XX.......X.......X.X...X...XXX......X..........X......] duration:
18.323512023s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:04:42 UTC)
orbiter warning: context deadline exceeded
Duration: 18.323638694s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.585690521s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:04:37 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.585713484s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.......XX.......X.......X.X...X...XXX......X..........X.....] duration:
20.879468839s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:04:16 UTC)
orbiter warning: context deadline exceeded
Duration: 20.879503234s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
5.742367204s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:04:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.742394366s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [........XX.......X.......X.X...X...XXX......X..........X....] duration:
20.279266126s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:03:53 UTC)
orbiter warning: context deadline exceeded
Duration: 20.27928344s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001475645s trace:
https://jaeger.dev.storj.io/trace/3af67fb1d2270c3a
FAIL (2026-01-09 11:03:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001496884s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
5.829716757s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:03:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.829745371s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.601055897s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:03:26 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.601084492s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
5.782181522s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:03:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.782213814s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
5.698132802s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:03:09 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.698168008s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.........XX.......X.......X.X...X...XXX......X..........X...] duration:
21.228828748s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:03:01 UTC)
orbiter warning: context deadline exceeded
Duration: 21.228876229s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [.XXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001075444s trace:
https://jaeger.dev.storj.io/trace/5e3e1b42830e0953
FAIL (2026-01-09 11:02:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001092155s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..........XX.......X.......X.X...X...XXX......X..........X..] duration:
29.747977282s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:02:31 UTC)
orbiter warning: context deadline exceeded
Duration: 29.747994475s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
5.533573382s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:02:19 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.533602918s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
5.841752096s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:02:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.84178048s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [X.XXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00118243s trace:
https://jaeger.dev.storj.io/trace/18a6480c8809406d
FAIL (2026-01-09 11:02:02 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001204221s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X..........XX.......X.......X.X...X...XXX......X..........X.] duration:
27.825015197s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:02:01 UTC)
orbiter warning: context deadline exceeded
Duration: 27.825043459s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
5.678524788s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:01:46 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.678550847s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
5.771250484s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:01:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.771277774s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XX.XXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000984911s trace:
https://jaeger.dev.storj.io/trace/26593c01419f032c
FAIL (2026-01-09 11:01:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001186664s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.X..........XX.......X.......X.X...X...XXX......X..........X] duration:
30.001603831s trace:
https://jaeger.dev.storj.io/trace/7b81528b4b7ba7a3
FAIL (2026-01-09 11:01:21 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001625653s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.813454209s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 11:00:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.8134709s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
5.732646913s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 11:00:34 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.732694002s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXX.XXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001216503s trace:
https://jaeger.dev.storj.io/trace/302870cd6efb384b
FAIL (2026-01-09 11:00:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001240088s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..X..........XX.......X.......X.X...X...XXX......X..........] duration:
18.434157821s trace:
https://jaeger.dev.storj.io/trace/16d7ab6522716d43
FAIL (2026-01-09 11:00:24 UTC)
orbiter warning: context deadline exceeded
Duration: 18.434209991s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...X..........XX.......X.......X.X...X...XXX......X.........] duration:
23.610749742s trace:
https://jaeger.dev.storj.io/trace/16d7ab6522716d43
FAIL (2026-01-09 10:59:49 UTC)
orbiter warning: context deadline exceeded
Duration: 23.610777134s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXX.XXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001054446s trace:
https://jaeger.dev.storj.io/trace/36c012cf8ddce4f4
FAIL (2026-01-09 10:59:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001089602s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
5.629607078s trace:
https://jaeger.dev.storj.io/trace/5637bf42dd3ae530
FAIL (2026-01-09 10:59:36 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
FACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355190787 (835h20m47.226273832s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355202760 (835h20m47.226285806s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355514103 (835h20m47.226597138s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355543229 (835h20m47.226626264s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355815477 (835h20m47.226898522s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.355822010 (835h20m47.226905045s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356121189 (835h20m47.227204234s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356156556 (835h20m47.227239601s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356418776 (835h20m47.227501821s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356471616 (835h20m47.227554651s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356805663 (835h20m47.227888698s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.356924218 (835h20m47.228007253s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357129028 (835h20m47.228212063s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357224250 (835h20m47.228307285s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.357456683 (835h20m47.228539718s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:47.358609427 (835h20m47.229692463s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Uploading piece...
Jan 9 10:49:48.775784669 (835h20m48.646867714s) [upload=730683]: Next returned nil segment
Jan 9 10:49:48.775796972 (835h20m48.646880007s) [upload=730683]: Waiting for error group managing segments...
Jan 9 10:49:49.176111669 (835h20m49.047194704s) [noise=true , address=64.49.13.169:4711 , piece_id=PAMZR4LAJ7UBOOJFOQSLKMEIFFBQW3WR6T6ZKF4RVT4ENZBGQSWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.381724343 (835h20m49.252807388s) [noise=true , address=145.79.227.12:4772 , piece_id=TOBDTPLX6KNS7KKYEN32BB3UP5LSTBBQI6DISKQSMZR6IMDY43BA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.406288056 (835h20m49.277371091s) [noise=true , address=77.90.31.19:4859 , piece_id=MCDKT4LRKT4SUK6XS7RSAKB5OY5XJBT4OMHJX4R7N23ZWNHB65PA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.431670719 (835h20m49.302753774s) [noise=true , address=145.79.227.49:4341 , piece_id=SX2OPLTAX2274VYVWFACYN5W76CJAPFJWPT2U4PHNHESAXZMWGAA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.436116914 (835h20m49.307199949s) [noise=true , address=145.79.227.29:4974 , piece_id=BGK4M4PJMATFMQXVCI6RAZXTBRMQO75XVLZTPUY464CIVMIINQ3Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.447697075 (835h20m49.31878011s) [noise=true , address=145.79.227.40:4079 , piece_id=ON6ZIKANQRHQ2RWLRH5FJKMK5Z7FOFVJSPZKA5H2UXMS7SDAKCLA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.450386596 (835h20m49.321469632s) [noise=true , address=77.90.31.13:4631 , piece_id=GA3QERGB3XRNEG2IPZ4IJEY4RSYJDNGIL4OTMP2WRXN4UL6YUBBQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460009109 (835h20m49.331092144s) [noise=true , address=145.79.227.34:4915 , piece_id=5BFWEY66KKDSYFLFGEWICJD5RGH7IG3LCQAFACDVHE2CSEFW5MZQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.460651291 (835h20m49.331734326s) [noise=true , address=145.79.227.3:4176 , piece_id=MQYGRJ5JHRP3YOUPU5UOMDOW5I7NTC6LHFODLSE7DD4HRR2QVVKA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.467339557 (835h20m49.338422602s) [noise=true , address=145.79.227.30:4788 , piece_id=LVXP447NSTWZRNWJWX3MAEHGGM6FEAXBXBPUNKLQMQKTCCBBASEQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.717024371 (835h20m49.588107406s) [noise=true , address=77.90.31.11:4563 , piece_id=22AAVEB2QRVXUPP3G7FVSRUSLFMBU5UXXWHMYXIJR5AFTX2WMOWQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:49.968229030 (835h20m49.839312065s) [noise=true , address=45.91.116.9:4273 , piece_id=CIF6IAT4QKIEMIHSC54ZFIHKJOFHNNYVETKRCRGLRH2BITNLW54Q, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.067335843 (835h20m49.938418878s) [noise=true , address=145.79.227.35:4497 , piece_id=IBQYN5AII4NW2VT3EAKG7N2T6W7JYYJRV4TGZ57EQYU3FGB62UFQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084374528 (835h20m49.955457573s) [noise=true , address=193.149.164.143:4695 , piece_id=RC27R5N6E657KXQE6T7YV4UHHZ6P3WX64WWCXVKMMCDCTP5BDLVA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: <nil>
Jan 9 10:49:50.084395047 (835h20m49.955478082s) [seg_pos={0 0}, upload=730683]: Segment reached optimal threshold of 14 pieces.
Jan 9 10:49:50.084894708 (835h20m49.955977773s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.084913094 (835h20m49.955996129s) [noise=true , address=145.79.227.33:4903 , piece_id=2OBP6YKWVNKH6KA4H4FLPNB4A2B32X6PPOBN2RSOJKGLG3J5KJHA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085306763 (835h20m49.956389808s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085316281 (835h20m49.956399326s) [noise=true , address=145.79.227.43:4177 , piece_id=WKT6UM7VC5K4VHCZDFALV64A6CMZSKQIHRJTJGEJ6KYI5THAZBAQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085450537 (835h20m49.956533582s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju): protocol: expected piece hash; context canceled; EOF
Jan 9 10:49:50.085456017 (835h20m49.956539052s) [noise=true , address=45.91.116.14:4203 , piece_id=7CHAHCEYNDYLK7W5RDDZFRO5L6DM6GALQMEEFDMK4U5ECBTR3OIA, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085651860 (835h20m49.956734895s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m3Lw8yL9wVq38nW514iaLooMVgikbQBGA9vrFsr66jx5w1xXj): protocol: read tcp 188.166.225.170:43994->45.91.116.5:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 9 10:49:50.085659655 (835h20m49.95674269s) [noise=true , address=45.91.116.5:4321 , piece_id=O36WN7CMFDT2NOV3Q67P2I3WOGQU2E3CZERHBIHOZIXWLCHRDTRQ, seg_pos={0 0}, upload=730683]: Long tail cancellation due to optimal threshold
Jan 9 10:49:50.085679282 (835h20m49.956762317s) [seg_pos={0 0}, upload=730683]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 9 10:49:50.085747111 (835h20m49.956830146s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:49:50.085752582 (835h20m49.956835617s) [upload=730683]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:49:50.378436022 (835h20m50.249519067s) [upload=730683]: Done uploading segments...
Jan 9 10:49:50.378458726 (835h20m50.249541781s) [upload=730683]: Upload finished. err: <nil>
Duration: 5.629635853s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.773772968s trace:
https://jaeger.dev.storj.io/trace/dc49b1b5ed04c9d
FAIL (2026-01-09 10:59:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q) context deadline exceeded
======LOGS======
Jan 9 10:33:22.955109470 (835h4m22.826192505s) [upload=730448]: Starting upload
Jan 9 10:33:22.955189192 (835h4m22.826272257s) [upload=730448]: Done starting upload
Jan 9 10:33:22.955294262 (835h4m22.826377297s) [upload=730448]: Uploading segments...
Jan 9 10:33:22.955467612 (835h4m22.826550667s) [upload=730448]: Got next segment. Inline: false
Jan 9 10:33:22.955516936 (835h4m22.826599971s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 9 10:33:22.955522687 (835h4m22.826605722s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 9 10:33:23.307370495 (835h4m23.17845353s) [seg_pos={0 0}, upload=730448]: Begin upload segment...
Jan 9 10:33:23.307423696 (835h4m23.178506731s) [seg_pos={0 0}, upload=730448]: Done begin upload segment.
Jan 9 10:33:23.307463803 (835h4m23.178546838s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.307538755 (835h4m23.17862179s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308001716 (835h4m23.179084761s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308313701 (835h4m23.179396736s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308635564 (835h4m23.179718599s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.308908484 (835h4m23.179991519s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309180651 (835h4m23.180263697s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309461997 (835h4m23.180545042s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.309765405 (835h4m23.18084844s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311739645 (835h4m23.18282269s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.311970835 (835h4m23.18305387s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.313619995 (835h4m23.18470303s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.315012797 (835h4m23.186095832s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.316157818 (835h4m23.187240853s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.317307697 (835h4m23.188390732s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:23.318458238 (835h4m23.189541273s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Uploading piece...
Jan 9 10:33:25.896810636 (835h4m25.767893671s) [upload=730448]: Next returned nil segment
Jan 9 10:33:25.896830434 (835h4m25.767913469s) [upload=730448]: Waiting for error group managing segments...
Jan 9 10:33:26.294921876 (835h4m26.166004911s) [noise=true , address=77.90.31.17:4770 , piece_id=SAWBSKOVKFGBDHIPUOZLBJPN73T32EC55WTQOQLR5DFBDNX7S6SQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.418908564 (835h4m26.289991599s) [noise=true , address=145.79.227.13:4802 , piece_id=UJTYIWBMCQFBQOW6ZCHUT7GFJZPA44UQNMQCAY6CUQVNQNKH3O4Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.441883322 (835h4m26.312966357s) [noise=true , address=45.91.116.7:4011 , piece_id=CLT4ZFM63P6F32U5LIBWO63EENA2VZO2LLLLT54N5TJNWOG2GCOQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:26.580106278 (835h4m26.451189313s) [noise=true , address=45.91.116.6:4974 , piece_id=PRBRDT7MGU224NRITBNRGWW4JYHIYT23T7UA5XAWQM3GXGJHVG2A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.069390134 (835h4m26.940473169s) [noise=true , address=145.79.227.5:4222 , piece_id=F7FES6SVIKUVEUD3LSK5GSR6AMOTTZUDSCSUXOYELMFUTKFR7KDQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.314962439 (835h4m27.186045484s) [noise=true , address=77.90.31.16:4735 , piece_id=NKAQRHSPER6AHDFOTY3JUBDMQMJX56D4SIFLBETRWS3PY3SHKBJQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.329732422 (835h4m27.200815457s) [noise=true , address=45.91.116.5:4297 , piece_id=DGJOA5SKDV4FUAN6PG3VF4IDEFCYYF4S2UE5XWKKA2IWZQWZAHRQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.360057426 (835h4m27.231140471s) [noise=true , address=145.79.227.50:4375 , piece_id=MIU5AR5XPH7LUT76QTMJ5UYGM6P7RBZBAZUDQVQGCOG5UHXAX55Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.391338318 (835h4m27.262421353s) [noise=true , address=145.79.227.14:4840 , piece_id=SNTLL22G5ODMBLDQAM5FYRVXWMPYHY2A7WFJ62EKYVQCF6TU6QRA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:27.418342067 (835h4m27.289425113s) [noise=true , address=145.79.227.45:4230 , piece_id=APMB2QIRJVDUXGWAWB6KDEWAWGWPTL47JVPOIABXPS7CFOONVN6Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.177000491 (835h4m28.048083526s) [noise=true , address=145.79.227.37:4567 , piece_id=JB4BJUUYDCVUGPI7D2IBT54NS4DBUEM4EMCJMOE24MMGY4INEEKQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.222148008 (835h4m28.093231043s) [noise=true , address=77.90.31.10:4519 , piece_id=M2GXUPUJSHS5N5CXWNUNN4FZ6FJABUDARJKUQPXMM76EP7OHP37Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509360190 (835h4m28.380443225s) [noise=true , address=145.79.227.18:4007 , piece_id=VW3HF2W2UHBWN5U54PMJIRAXGZ327H2OKUKGO6EGTITRQ6OHBKHQ, seg_pos={0 0}, upload=730448]: Done uploading piece. err: <nil>
Jan 9 10:33:28.509378345 (835h4m28.38046138s) [seg_pos={0 0}, upload=730448]: Segment reached optimal threshold of 13 pieces.
Jan 9 10:33:28.509578085 (835h4m28.38066112s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509586261 (835h4m28.380669296s) [noise=true , address=145.79.227.17:4972 , piece_id=BJOR74XPTTFTU7IRXZK67LRGV55CGMOCNMTATP2U6E5HRP7BDCQA, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509594857 (835h4m28.380677892s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509598895 (835h4m28.38068193s) [noise=true , address=45.91.116.4:4937 , piece_id=L5HSNVLOG3INJZYKBNTKJXRTBSDNJ5YIQCTEVHAEGBZTTXX3R43A, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509740835 (835h4m28.38082387s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Jan 9 10:33:28.509745914 (835h4m28.380828949s) [noise=true , address=145.79.227.39:4013 , piece_id=NUX433DGHMPD5T7SVDQCHDHCWNYNIFGEPR3MSUBYCE3ZKSMTRD5Q, seg_pos={0 0}, upload=730448]: Long tail cancellation due to optimal threshold
Jan 9 10:33:28.509782173 (835h4m28.380865208s) [seg_pos={0 0}, upload=730448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 9 10:33:28.509931849 (835h4m28.381014884s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 9 10:33:28.509937229 (835h4m28.381020264s) [upload=730448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 9 10:33:28.946569209 (835h4m28.817652244s) [upload=730448]: Done uploading segments...
Jan 9 10:33:28.946586792 (835h4m28.817669837s) [upload=730448]: Upload finished. err: <nil>
Duration: 5.773809026s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00080076s trace:
https://jaeger.dev.storj.io/trace/5842a9a9510fc045
FAIL (2026-01-09 10:59:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000829516s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [....X..........XX.......X.......X.X...X...XXX......X........] duration:
18.696859651s trace:
https://jaeger.dev.storj.io/trace/16d7ab6522716d43
FAIL (2026-01-09 10:59:07 UTC)
orbiter warning: context deadline exceeded
Duration: 18.69689095s
edge
gateway
us-select-1
|