Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 334167 reports (Page 111 of 6684)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX..X.....]
duration: 
5.639114781s
trace: 
https://jaeger.dev.storj.io/trace/709438126642b358
FAIL (2025-12-27 10:00:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ) context deadline exceeded
======LOGS======
ents...
Dec 27 09:56:17.447680280 (522h27m17.318763326s) [upload=458329]: Got next segment. Inline: false
Dec 27 09:56:17.447722972 (522h27m17.318806017s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:56:17.447729133 (522h27m17.318812168s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:56:17.690112332 (522h27m17.561195367s) [seg_pos={0 0}, upload=458329]: Begin upload segment...
Dec 27 09:56:17.690156355 (522h27m17.56123939s) [seg_pos={0 0}, upload=458329]: Done begin upload segment.
Dec 27 09:56:17.690181734 (522h27m17.561264769s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690281032 (522h27m17.561364097s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690749224 (522h27m17.561832269s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691078992 (522h27m17.562162037s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691420743 (522h27m17.562503778s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691693562 (522h27m17.562776607s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692053688 (522h27m17.563136733s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692341416 (522h27m17.563424461s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.693984435 (522h27m17.56506747s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.694374447 (522h27m17.565457492s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.695695984 (522h27m17.566779029s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.697014394 (522h27m17.568097439s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.698560087 (522h27m17.569643122s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.699370090 (522h27m17.570453135s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.701688484 (522h27m17.572771529s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.704419655 (522h27m17.57550269s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:19.422586774 (522h27m19.293669819s) [upload=458329]: Next returned nil segment
Dec 27 09:56:19.422609567 (522h27m19.293692633s) [upload=458329]: Waiting for error group managing segments...
Dec 27 09:56:19.698368098 (522h27m19.569451133s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.705627420 (522h27m19.576710455s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.716770139 (522h27m19.587853174s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.717020636 (522h27m19.588103671s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.730448624 (522h27m19.601531659s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.731351583 (522h27m19.602434618s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.736581261 (522h27m19.607664296s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.894799456 (522h27m19.765882501s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.910384272 (522h27m19.781467317s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.931710831 (522h27m19.802793876s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.967157407 (522h27m19.838240453s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.012355212 (522h27m19.883438257s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267763061 (522h27m20.138846096s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267792397 (522h27m20.138875442s) [seg_pos={0 0}, upload=458329]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:56:20.268023498 (522h27m20.139106543s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection
Dec 27 09:56:20.268034428 (522h27m20.139117473s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268117716 (522h27m20.139200751s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 27 09:56:20.268132946 (522h27m20.139215981s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268421725 (522h27m20.13950476s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection
Dec 27 09:56:20.268430482 (522h27m20.139513517s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268446914 (522h27m20.139529959s) [seg_pos={0 0}, upload=458329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:56:20.268527928 (522h27m20.139610963s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:56:20.268532236 (522h27m20.139615281s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:56:20.560043612 (522h27m20.431126648s) [upload=458329]: Done uploading segments...
Dec 27 09:56:20.560062399 (522h27m20.431145444s) [upload=458329]: Upload finished. err: <nil>
Duration: 5.639148023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [..X........X.............X........................X.X.......]
duration: 
32.372460114s
trace: 
https://jaeger.dev.storj.io/trace/d450b6ce193c728
FAIL (2025-12-27 10:00:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233076300 (522h24m11.104159345s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233549011 (522h24m11.104632056s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233994970 (522h24m11.105078025s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234029806 (522h24m11.105112842s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234387447 (522h24m11.105470482s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234549115 (522h24m11.10563215s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234743615 (522h24m11.10582666s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235123268 (522h24m11.106206303s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235636846 (522h24m11.106719891s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235999697 (522h24m11.107082772s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236444434 (522h24m11.107527469s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236878911 (522h24m11.107961956s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237221313 (522h24m11.108304348s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237993243 (522h24m11.109076278s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.246722595 (522h24m11.11780568s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:12.530330220 (522h24m12.401413255s) [upload=458284]: Next returned nil segment
Dec 27 09:53:12.530349887 (522h24m12.401432922s) [upload=458284]: Waiting for error group managing segments...
Dec 27 09:53:12.876532286 (522h24m12.747615331s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:12.965370209 (522h24m12.836453244s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.115211782 (522h24m12.986294817s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.140777494 (522h24m13.011860529s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.169081760 (522h24m13.040164805s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.280115036 (522h24m13.151198071s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.293944019 (522h24m13.165027094s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.299058658 (522h24m13.170141693s) [noise=true , address=77.90.31.14:4678     , piece_id=N2A4RJF5FFH3Y7J6EO4IUJ6HHVJBODEIRKMNUQIXHLTXUAHOMDCA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.388158077 (522h24m13.259241123s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.480680246 (522h24m13.351763281s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.691531151 (522h24m13.562614196s) [noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.976152859 (522h24m13.847235904s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.980566223 (522h24m13.851649258s) [noise=true , address=145.79.227.30:4777   , piece_id=FZ55XXJO5UC4SYAOX5PBHKD2FQG43WDEJDQGAWTMAYVJ2XQWBKMQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054854099 (522h24m13.925937134s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054871502 (522h24m13.925954537s) [seg_pos={0 0}, upload=458284]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:53:14.055044472 (522h24m13.926127507s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055051375 (522h24m13.92613441s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055334113 (522h24m13.926417148s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection
Dec 27 09:53:14.055340785 (522h24m13.926423831s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055487816 (522h24m13.926570851s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055494018 (522h24m13.926577053s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055610760 (522h24m13.926693795s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055616471 (522h24m13.926699516s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055630578 (522h24m13.926713613s) [seg_pos={0 0}, upload=458284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:53:14.055757609 (522h24m13.926840644s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:14.055762910 (522h24m13.926845945s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:14.343772328 (522h24m14.214855373s) [upload=458284]: Done uploading segments...
Dec 27 09:53:14.343848974 (522h24m14.214932009s) [upload=458284]: Upload finished. err: <nil>
Duration: 32.372489409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [.XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX..X....]
duration: 
5.676290299s
trace: 
https://jaeger.dev.storj.io/trace/709438126642b358
FAIL (2025-12-27 09:59:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ) context deadline exceeded
======LOGS======
ents...
Dec 27 09:56:17.447680280 (522h27m17.318763326s) [upload=458329]: Got next segment. Inline: false
Dec 27 09:56:17.447722972 (522h27m17.318806017s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:56:17.447729133 (522h27m17.318812168s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:56:17.690112332 (522h27m17.561195367s) [seg_pos={0 0}, upload=458329]: Begin upload segment...
Dec 27 09:56:17.690156355 (522h27m17.56123939s) [seg_pos={0 0}, upload=458329]: Done begin upload segment.
Dec 27 09:56:17.690181734 (522h27m17.561264769s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690281032 (522h27m17.561364097s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690749224 (522h27m17.561832269s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691078992 (522h27m17.562162037s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691420743 (522h27m17.562503778s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691693562 (522h27m17.562776607s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692053688 (522h27m17.563136733s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692341416 (522h27m17.563424461s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.693984435 (522h27m17.56506747s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.694374447 (522h27m17.565457492s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.695695984 (522h27m17.566779029s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.697014394 (522h27m17.568097439s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.698560087 (522h27m17.569643122s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.699370090 (522h27m17.570453135s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.701688484 (522h27m17.572771529s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.704419655 (522h27m17.57550269s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:19.422586774 (522h27m19.293669819s) [upload=458329]: Next returned nil segment
Dec 27 09:56:19.422609567 (522h27m19.293692633s) [upload=458329]: Waiting for error group managing segments...
Dec 27 09:56:19.698368098 (522h27m19.569451133s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.705627420 (522h27m19.576710455s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.716770139 (522h27m19.587853174s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.717020636 (522h27m19.588103671s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.730448624 (522h27m19.601531659s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.731351583 (522h27m19.602434618s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.736581261 (522h27m19.607664296s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.894799456 (522h27m19.765882501s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.910384272 (522h27m19.781467317s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.931710831 (522h27m19.802793876s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.967157407 (522h27m19.838240453s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.012355212 (522h27m19.883438257s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267763061 (522h27m20.138846096s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267792397 (522h27m20.138875442s) [seg_pos={0 0}, upload=458329]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:56:20.268023498 (522h27m20.139106543s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection
Dec 27 09:56:20.268034428 (522h27m20.139117473s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268117716 (522h27m20.139200751s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 27 09:56:20.268132946 (522h27m20.139215981s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268421725 (522h27m20.13950476s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection
Dec 27 09:56:20.268430482 (522h27m20.139513517s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268446914 (522h27m20.139529959s) [seg_pos={0 0}, upload=458329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:56:20.268527928 (522h27m20.139610963s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:56:20.268532236 (522h27m20.139615281s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:56:20.560043612 (522h27m20.431126648s) [upload=458329]: Done uploading segments...
Dec 27 09:56:20.560062399 (522h27m20.431145444s) [upload=458329]: Upload finished. err: <nil>
Duration: 5.676310909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [...X........X.............X........................X.X......]
duration: 
41.348299036s
trace: 
https://jaeger.dev.storj.io/trace/d450b6ce193c728
FAIL (2025-12-27 09:59:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233076300 (522h24m11.104159345s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233549011 (522h24m11.104632056s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233994970 (522h24m11.105078025s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234029806 (522h24m11.105112842s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234387447 (522h24m11.105470482s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234549115 (522h24m11.10563215s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234743615 (522h24m11.10582666s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235123268 (522h24m11.106206303s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235636846 (522h24m11.106719891s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235999697 (522h24m11.107082772s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236444434 (522h24m11.107527469s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236878911 (522h24m11.107961956s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237221313 (522h24m11.108304348s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237993243 (522h24m11.109076278s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.246722595 (522h24m11.11780568s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:12.530330220 (522h24m12.401413255s) [upload=458284]: Next returned nil segment
Dec 27 09:53:12.530349887 (522h24m12.401432922s) [upload=458284]: Waiting for error group managing segments...
Dec 27 09:53:12.876532286 (522h24m12.747615331s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:12.965370209 (522h24m12.836453244s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.115211782 (522h24m12.986294817s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.140777494 (522h24m13.011860529s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.169081760 (522h24m13.040164805s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.280115036 (522h24m13.151198071s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.293944019 (522h24m13.165027094s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.299058658 (522h24m13.170141693s) [noise=true , address=77.90.31.14:4678     , piece_id=N2A4RJF5FFH3Y7J6EO4IUJ6HHVJBODEIRKMNUQIXHLTXUAHOMDCA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.388158077 (522h24m13.259241123s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.480680246 (522h24m13.351763281s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.691531151 (522h24m13.562614196s) [noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.976152859 (522h24m13.847235904s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.980566223 (522h24m13.851649258s) [noise=true , address=145.79.227.30:4777   , piece_id=FZ55XXJO5UC4SYAOX5PBHKD2FQG43WDEJDQGAWTMAYVJ2XQWBKMQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054854099 (522h24m13.925937134s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054871502 (522h24m13.925954537s) [seg_pos={0 0}, upload=458284]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:53:14.055044472 (522h24m13.926127507s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055051375 (522h24m13.92613441s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055334113 (522h24m13.926417148s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection
Dec 27 09:53:14.055340785 (522h24m13.926423831s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055487816 (522h24m13.926570851s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055494018 (522h24m13.926577053s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055610760 (522h24m13.926693795s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055616471 (522h24m13.926699516s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055630578 (522h24m13.926713613s) [seg_pos={0 0}, upload=458284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:53:14.055757609 (522h24m13.926840644s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:14.055762910 (522h24m13.926845945s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:14.343772328 (522h24m14.214855373s) [upload=458284]: Done uploading segments...
Dec 27 09:53:14.343848974 (522h24m14.214932009s) [upload=458284]: Upload finished. err: <nil>
Duration: 41.348327851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX..X...]
duration: 
5.621438922s
trace: 
https://jaeger.dev.storj.io/trace/709438126642b358
FAIL (2025-12-27 09:58:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ) context deadline exceeded
======LOGS======
ents...
Dec 27 09:56:17.447680280 (522h27m17.318763326s) [upload=458329]: Got next segment. Inline: false
Dec 27 09:56:17.447722972 (522h27m17.318806017s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:56:17.447729133 (522h27m17.318812168s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:56:17.690112332 (522h27m17.561195367s) [seg_pos={0 0}, upload=458329]: Begin upload segment...
Dec 27 09:56:17.690156355 (522h27m17.56123939s) [seg_pos={0 0}, upload=458329]: Done begin upload segment.
Dec 27 09:56:17.690181734 (522h27m17.561264769s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690281032 (522h27m17.561364097s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690749224 (522h27m17.561832269s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691078992 (522h27m17.562162037s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691420743 (522h27m17.562503778s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691693562 (522h27m17.562776607s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692053688 (522h27m17.563136733s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692341416 (522h27m17.563424461s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.693984435 (522h27m17.56506747s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.694374447 (522h27m17.565457492s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.695695984 (522h27m17.566779029s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.697014394 (522h27m17.568097439s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.698560087 (522h27m17.569643122s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.699370090 (522h27m17.570453135s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.701688484 (522h27m17.572771529s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.704419655 (522h27m17.57550269s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:19.422586774 (522h27m19.293669819s) [upload=458329]: Next returned nil segment
Dec 27 09:56:19.422609567 (522h27m19.293692633s) [upload=458329]: Waiting for error group managing segments...
Dec 27 09:56:19.698368098 (522h27m19.569451133s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.705627420 (522h27m19.576710455s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.716770139 (522h27m19.587853174s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.717020636 (522h27m19.588103671s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.730448624 (522h27m19.601531659s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.731351583 (522h27m19.602434618s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.736581261 (522h27m19.607664296s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.894799456 (522h27m19.765882501s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.910384272 (522h27m19.781467317s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.931710831 (522h27m19.802793876s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.967157407 (522h27m19.838240453s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.012355212 (522h27m19.883438257s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267763061 (522h27m20.138846096s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267792397 (522h27m20.138875442s) [seg_pos={0 0}, upload=458329]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:56:20.268023498 (522h27m20.139106543s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection
Dec 27 09:56:20.268034428 (522h27m20.139117473s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268117716 (522h27m20.139200751s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 27 09:56:20.268132946 (522h27m20.139215981s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268421725 (522h27m20.13950476s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection
Dec 27 09:56:20.268430482 (522h27m20.139513517s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268446914 (522h27m20.139529959s) [seg_pos={0 0}, upload=458329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:56:20.268527928 (522h27m20.139610963s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:56:20.268532236 (522h27m20.139615281s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:56:20.560043612 (522h27m20.431126648s) [upload=458329]: Done uploading segments...
Dec 27 09:56:20.560062399 (522h27m20.431145444s) [upload=458329]: Upload finished. err: <nil>
Duration: 5.62146902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX..X..]
duration: 
6.035688452s
trace: 
https://jaeger.dev.storj.io/trace/709438126642b358
FAIL (2025-12-27 09:57:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ) context deadline exceeded
======LOGS======
ents...
Dec 27 09:56:17.447680280 (522h27m17.318763326s) [upload=458329]: Got next segment. Inline: false
Dec 27 09:56:17.447722972 (522h27m17.318806017s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:56:17.447729133 (522h27m17.318812168s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:56:17.690112332 (522h27m17.561195367s) [seg_pos={0 0}, upload=458329]: Begin upload segment...
Dec 27 09:56:17.690156355 (522h27m17.56123939s) [seg_pos={0 0}, upload=458329]: Done begin upload segment.
Dec 27 09:56:17.690181734 (522h27m17.561264769s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690281032 (522h27m17.561364097s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690749224 (522h27m17.561832269s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691078992 (522h27m17.562162037s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691420743 (522h27m17.562503778s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691693562 (522h27m17.562776607s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692053688 (522h27m17.563136733s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692341416 (522h27m17.563424461s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.693984435 (522h27m17.56506747s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.694374447 (522h27m17.565457492s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.695695984 (522h27m17.566779029s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.697014394 (522h27m17.568097439s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.698560087 (522h27m17.569643122s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.699370090 (522h27m17.570453135s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.701688484 (522h27m17.572771529s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.704419655 (522h27m17.57550269s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:19.422586774 (522h27m19.293669819s) [upload=458329]: Next returned nil segment
Dec 27 09:56:19.422609567 (522h27m19.293692633s) [upload=458329]: Waiting for error group managing segments...
Dec 27 09:56:19.698368098 (522h27m19.569451133s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.705627420 (522h27m19.576710455s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.716770139 (522h27m19.587853174s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.717020636 (522h27m19.588103671s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.730448624 (522h27m19.601531659s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.731351583 (522h27m19.602434618s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.736581261 (522h27m19.607664296s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.894799456 (522h27m19.765882501s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.910384272 (522h27m19.781467317s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.931710831 (522h27m19.802793876s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.967157407 (522h27m19.838240453s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.012355212 (522h27m19.883438257s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267763061 (522h27m20.138846096s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267792397 (522h27m20.138875442s) [seg_pos={0 0}, upload=458329]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:56:20.268023498 (522h27m20.139106543s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection
Dec 27 09:56:20.268034428 (522h27m20.139117473s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268117716 (522h27m20.139200751s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 27 09:56:20.268132946 (522h27m20.139215981s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268421725 (522h27m20.13950476s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection
Dec 27 09:56:20.268430482 (522h27m20.139513517s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268446914 (522h27m20.139529959s) [seg_pos={0 0}, upload=458329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:56:20.268527928 (522h27m20.139610963s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:56:20.268532236 (522h27m20.139615281s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:56:20.560043612 (522h27m20.431126648s) [upload=458329]: Done uploading segments...
Dec 27 09:56:20.560062399 (522h27m20.431145444s) [upload=458329]: Upload finished. err: <nil>
Duration: 6.035719852s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [....X........X.............X........................X.X.....]
duration: 
27.84940118s
trace: 
https://jaeger.dev.storj.io/trace/d450b6ce193c728
FAIL (2025-12-27 09:57:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233076300 (522h24m11.104159345s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233549011 (522h24m11.104632056s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233994970 (522h24m11.105078025s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234029806 (522h24m11.105112842s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234387447 (522h24m11.105470482s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234549115 (522h24m11.10563215s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234743615 (522h24m11.10582666s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235123268 (522h24m11.106206303s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235636846 (522h24m11.106719891s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235999697 (522h24m11.107082772s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236444434 (522h24m11.107527469s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236878911 (522h24m11.107961956s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237221313 (522h24m11.108304348s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237993243 (522h24m11.109076278s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.246722595 (522h24m11.11780568s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:12.530330220 (522h24m12.401413255s) [upload=458284]: Next returned nil segment
Dec 27 09:53:12.530349887 (522h24m12.401432922s) [upload=458284]: Waiting for error group managing segments...
Dec 27 09:53:12.876532286 (522h24m12.747615331s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:12.965370209 (522h24m12.836453244s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.115211782 (522h24m12.986294817s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.140777494 (522h24m13.011860529s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.169081760 (522h24m13.040164805s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.280115036 (522h24m13.151198071s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.293944019 (522h24m13.165027094s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.299058658 (522h24m13.170141693s) [noise=true , address=77.90.31.14:4678     , piece_id=N2A4RJF5FFH3Y7J6EO4IUJ6HHVJBODEIRKMNUQIXHLTXUAHOMDCA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.388158077 (522h24m13.259241123s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.480680246 (522h24m13.351763281s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.691531151 (522h24m13.562614196s) [noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.976152859 (522h24m13.847235904s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.980566223 (522h24m13.851649258s) [noise=true , address=145.79.227.30:4777   , piece_id=FZ55XXJO5UC4SYAOX5PBHKD2FQG43WDEJDQGAWTMAYVJ2XQWBKMQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054854099 (522h24m13.925937134s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054871502 (522h24m13.925954537s) [seg_pos={0 0}, upload=458284]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:53:14.055044472 (522h24m13.926127507s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055051375 (522h24m13.92613441s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055334113 (522h24m13.926417148s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection
Dec 27 09:53:14.055340785 (522h24m13.926423831s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055487816 (522h24m13.926570851s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055494018 (522h24m13.926577053s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055610760 (522h24m13.926693795s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055616471 (522h24m13.926699516s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055630578 (522h24m13.926713613s) [seg_pos={0 0}, upload=458284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:53:14.055757609 (522h24m13.926840644s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:14.055762910 (522h24m13.926845945s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:14.343772328 (522h24m14.214855373s) [upload=458284]: Done uploading segments...
Dec 27 09:53:14.343848974 (522h24m14.214932009s) [upload=458284]: Upload finished. err: <nil>
Duration: 27.849415607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [.....X........X.............X........................X.X....]
duration: 
49.434150498s
trace: 
https://jaeger.dev.storj.io/trace/d450b6ce193c728
FAIL (2025-12-27 09:57:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233076300 (522h24m11.104159345s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233549011 (522h24m11.104632056s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233994970 (522h24m11.105078025s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234029806 (522h24m11.105112842s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234387447 (522h24m11.105470482s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234549115 (522h24m11.10563215s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234743615 (522h24m11.10582666s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235123268 (522h24m11.106206303s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235636846 (522h24m11.106719891s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235999697 (522h24m11.107082772s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236444434 (522h24m11.107527469s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236878911 (522h24m11.107961956s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237221313 (522h24m11.108304348s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237993243 (522h24m11.109076278s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.246722595 (522h24m11.11780568s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:12.530330220 (522h24m12.401413255s) [upload=458284]: Next returned nil segment
Dec 27 09:53:12.530349887 (522h24m12.401432922s) [upload=458284]: Waiting for error group managing segments...
Dec 27 09:53:12.876532286 (522h24m12.747615331s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:12.965370209 (522h24m12.836453244s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.115211782 (522h24m12.986294817s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.140777494 (522h24m13.011860529s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.169081760 (522h24m13.040164805s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.280115036 (522h24m13.151198071s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.293944019 (522h24m13.165027094s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.299058658 (522h24m13.170141693s) [noise=true , address=77.90.31.14:4678     , piece_id=N2A4RJF5FFH3Y7J6EO4IUJ6HHVJBODEIRKMNUQIXHLTXUAHOMDCA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.388158077 (522h24m13.259241123s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.480680246 (522h24m13.351763281s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.691531151 (522h24m13.562614196s) [noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.976152859 (522h24m13.847235904s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.980566223 (522h24m13.851649258s) [noise=true , address=145.79.227.30:4777   , piece_id=FZ55XXJO5UC4SYAOX5PBHKD2FQG43WDEJDQGAWTMAYVJ2XQWBKMQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054854099 (522h24m13.925937134s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054871502 (522h24m13.925954537s) [seg_pos={0 0}, upload=458284]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:53:14.055044472 (522h24m13.926127507s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055051375 (522h24m13.92613441s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055334113 (522h24m13.926417148s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection
Dec 27 09:53:14.055340785 (522h24m13.926423831s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055487816 (522h24m13.926570851s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055494018 (522h24m13.926577053s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055610760 (522h24m13.926693795s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055616471 (522h24m13.926699516s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055630578 (522h24m13.926713613s) [seg_pos={0 0}, upload=458284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:53:14.055757609 (522h24m13.926840644s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:14.055762910 (522h24m13.926845945s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:14.343772328 (522h24m14.214855373s) [upload=458284]: Done uploading segments...
Dec 27 09:53:14.343848974 (522h24m14.214932009s) [upload=458284]: Upload finished. err: <nil>
Duration: 49.434166277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX..X.]
duration: 
5.830407737s
trace: 
https://jaeger.dev.storj.io/trace/709438126642b358
FAIL (2025-12-27 09:57:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ) context deadline exceeded
======LOGS======
ents...
Dec 27 09:56:17.447680280 (522h27m17.318763326s) [upload=458329]: Got next segment. Inline: false
Dec 27 09:56:17.447722972 (522h27m17.318806017s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:56:17.447729133 (522h27m17.318812168s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:56:17.690112332 (522h27m17.561195367s) [seg_pos={0 0}, upload=458329]: Begin upload segment...
Dec 27 09:56:17.690156355 (522h27m17.56123939s) [seg_pos={0 0}, upload=458329]: Done begin upload segment.
Dec 27 09:56:17.690181734 (522h27m17.561264769s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690281032 (522h27m17.561364097s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690749224 (522h27m17.561832269s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691078992 (522h27m17.562162037s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691420743 (522h27m17.562503778s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691693562 (522h27m17.562776607s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692053688 (522h27m17.563136733s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692341416 (522h27m17.563424461s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.693984435 (522h27m17.56506747s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.694374447 (522h27m17.565457492s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.695695984 (522h27m17.566779029s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.697014394 (522h27m17.568097439s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.698560087 (522h27m17.569643122s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.699370090 (522h27m17.570453135s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.701688484 (522h27m17.572771529s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.704419655 (522h27m17.57550269s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:19.422586774 (522h27m19.293669819s) [upload=458329]: Next returned nil segment
Dec 27 09:56:19.422609567 (522h27m19.293692633s) [upload=458329]: Waiting for error group managing segments...
Dec 27 09:56:19.698368098 (522h27m19.569451133s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.705627420 (522h27m19.576710455s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.716770139 (522h27m19.587853174s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.717020636 (522h27m19.588103671s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.730448624 (522h27m19.601531659s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.731351583 (522h27m19.602434618s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.736581261 (522h27m19.607664296s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.894799456 (522h27m19.765882501s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.910384272 (522h27m19.781467317s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.931710831 (522h27m19.802793876s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.967157407 (522h27m19.838240453s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.012355212 (522h27m19.883438257s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267763061 (522h27m20.138846096s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267792397 (522h27m20.138875442s) [seg_pos={0 0}, upload=458329]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:56:20.268023498 (522h27m20.139106543s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection
Dec 27 09:56:20.268034428 (522h27m20.139117473s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268117716 (522h27m20.139200751s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 27 09:56:20.268132946 (522h27m20.139215981s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268421725 (522h27m20.13950476s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection
Dec 27 09:56:20.268430482 (522h27m20.139513517s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268446914 (522h27m20.139529959s) [seg_pos={0 0}, upload=458329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:56:20.268527928 (522h27m20.139610963s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:56:20.268532236 (522h27m20.139615281s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:56:20.560043612 (522h27m20.431126648s) [upload=458329]: Done uploading segments...
Dec 27 09:56:20.560062399 (522h27m20.431145444s) [upload=458329]: Upload finished. err: <nil>
Duration: 5.83043597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX..X]
duration: 
30.002858375s
trace: 
https://jaeger.dev.storj.io/trace/709438126642b358
FAIL (2025-12-27 09:56:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ) context deadline exceeded
======LOGS======
ents...
Dec 27 09:56:17.447680280 (522h27m17.318763326s) [upload=458329]: Got next segment. Inline: false
Dec 27 09:56:17.447722972 (522h27m17.318806017s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:56:17.447729133 (522h27m17.318812168s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:56:17.690112332 (522h27m17.561195367s) [seg_pos={0 0}, upload=458329]: Begin upload segment...
Dec 27 09:56:17.690156355 (522h27m17.56123939s) [seg_pos={0 0}, upload=458329]: Done begin upload segment.
Dec 27 09:56:17.690181734 (522h27m17.561264769s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690281032 (522h27m17.561364097s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.690749224 (522h27m17.561832269s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691078992 (522h27m17.562162037s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691420743 (522h27m17.562503778s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.691693562 (522h27m17.562776607s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692053688 (522h27m17.563136733s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.692341416 (522h27m17.563424461s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.693984435 (522h27m17.56506747s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.694374447 (522h27m17.565457492s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.695695984 (522h27m17.566779029s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.697014394 (522h27m17.568097439s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.698560087 (522h27m17.569643122s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.699370090 (522h27m17.570453135s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.701688484 (522h27m17.572771529s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:17.704419655 (522h27m17.57550269s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Uploading piece...
Dec 27 09:56:19.422586774 (522h27m19.293669819s) [upload=458329]: Next returned nil segment
Dec 27 09:56:19.422609567 (522h27m19.293692633s) [upload=458329]: Waiting for error group managing segments...
Dec 27 09:56:19.698368098 (522h27m19.569451133s) [noise=true , address=145.79.227.14:4840   , piece_id=ZYZ4JRYGXLYN3K54S6ICU76NYYSX655IO546VDCIQFY2SUU676YA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.705627420 (522h27m19.576710455s) [noise=true , address=145.79.227.15:4874   , piece_id=KR4S2AWEVE3F4U5GMHIQ2AH2B7GHC3XCJCQIGYISCZAJ3QD4A4XQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.716770139 (522h27m19.587853174s) [noise=true , address=145.79.227.16:4911   , piece_id=3G2ADEVHDHEK7YRGEBO32QIU3N35CJRFZP4YAT5OSLPKNQRR5NGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.717020636 (522h27m19.588103671s) [noise=true , address=145.79.227.51:4411   , piece_id=XYEX3FI3IRT4YEBW7FG53QDTP6DP7DUVVHGHG4ZIQDMS7TQ5B4XA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.730448624 (522h27m19.601531659s) [noise=true , address=145.79.227.29:4943   , piece_id=C3NOM3CJM2QT332YERFVIQVOSN4AGGFXMSFRLY3S5WWE4XGVEZYQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.731351583 (522h27m19.602434618s) [noise=true , address=145.79.227.36:4518   , piece_id=SNUHX35ZBRGAJL3TCCNGY4SWCHES2CPT4TGUIG3ZRGRNCXMFE7VQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.736581261 (522h27m19.607664296s) [noise=true , address=145.79.227.13:4804   , piece_id=ZUGSWV4B3OUKDY64PHNK4R7H655NC74UKJQJLUPDQEFZ6EPDOWGA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.894799456 (522h27m19.765882501s) [noise=true , address=77.90.31.21:4877     , piece_id=ZCYAGXTYQDVLLKSS6YTNEFEP7AI7BEYVLLA4H6OF4JHRAHQUQHOQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.910384272 (522h27m19.781467317s) [noise=true , address=77.90.31.19:4841     , piece_id=ZZEPVS2MOI7YPJVCQLYAOYTUHIB6VM4GLIQRY3O63UU2LF7BUABQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.931710831 (522h27m19.802793876s) [noise=true , address=77.90.31.16:4736     , piece_id=EPUY3CL7WADIDK4PNWUSJS6JUDHDJJ4S7YCL5APHHZXJ376IJGGQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:19.967157407 (522h27m19.838240453s) [noise=true , address=45.91.116.5:4299     , piece_id=3AML6CST6GNMTZDK6KAR2EACGQRHH3V3X5GCHMTYDMR3MO7M327Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.012355212 (522h27m19.883438257s) [noise=true , address=45.91.116.4:4940     , piece_id=YGQJTSJAHSVGELQ25RHDI7DFKXKVNDWCX66TK3R6EMLCOM6CVBOA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267763061 (522h27m20.138846096s) [noise=true , address=45.91.116.13:4155    , piece_id=OEOUY2YI6ASN4WOB4TXPUHI7Z6BZLS5QP6EGDARI4Q6CPQZE4JLQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: <nil>
Dec 27 09:56:20.267792397 (522h27m20.138875442s) [seg_pos={0 0}, upload=458329]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:56:20.268023498 (522h27m20.139106543s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35778->45.91.116.10:4083: use of closed network connection
Dec 27 09:56:20.268034428 (522h27m20.139117473s) [noise=true , address=45.91.116.10:4083    , piece_id=2M2VHL2T4FVT444QO5O7KKIYIEGUKWO5MOTLCNLWQSJTKSYEDWFA, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268117716 (522h27m20.139200751s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 27 09:56:20.268132946 (522h27m20.139215981s) [noise=true , address=145.79.227.40:4051   , piece_id=N35IPPGOYUFNBHP6UWWWTCZVVNFJGVF5YYC6OUVMMSGGYYIIY3JQ, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268421725 (522h27m20.13950476s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56882->77.90.31.14:4661: use of closed network connection
Dec 27 09:56:20.268430482 (522h27m20.139513517s) [noise=true , address=77.90.31.14:4661     , piece_id=4VEPFW52QGAW5J27MMDTLQC7MJU2ULRKAWXJYEJQVYSUWW47I43Q, seg_pos={0 0}, upload=458329]: Long tail cancellation due to optimal threshold
Dec 27 09:56:20.268446914 (522h27m20.139529959s) [seg_pos={0 0}, upload=458329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:56:20.268527928 (522h27m20.139610963s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:56:20.268532236 (522h27m20.139615281s) [upload=458329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:56:20.560043612 (522h27m20.431126648s) [upload=458329]: Done uploading segments...
Dec 27 09:56:20.560062399 (522h27m20.431145444s) [upload=458329]: Upload finished. err: <nil>
Duration: 30.002881007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [......X........X.............X........................X.X...]
duration: 
36.727737201s
trace: 
https://jaeger.dev.storj.io/trace/d450b6ce193c728
FAIL (2025-12-27 09:56:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233076300 (522h24m11.104159345s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233549011 (522h24m11.104632056s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233994970 (522h24m11.105078025s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234029806 (522h24m11.105112842s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234387447 (522h24m11.105470482s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234549115 (522h24m11.10563215s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234743615 (522h24m11.10582666s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235123268 (522h24m11.106206303s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235636846 (522h24m11.106719891s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235999697 (522h24m11.107082772s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236444434 (522h24m11.107527469s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236878911 (522h24m11.107961956s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237221313 (522h24m11.108304348s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237993243 (522h24m11.109076278s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.246722595 (522h24m11.11780568s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:12.530330220 (522h24m12.401413255s) [upload=458284]: Next returned nil segment
Dec 27 09:53:12.530349887 (522h24m12.401432922s) [upload=458284]: Waiting for error group managing segments...
Dec 27 09:53:12.876532286 (522h24m12.747615331s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:12.965370209 (522h24m12.836453244s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.115211782 (522h24m12.986294817s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.140777494 (522h24m13.011860529s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.169081760 (522h24m13.040164805s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.280115036 (522h24m13.151198071s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.293944019 (522h24m13.165027094s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.299058658 (522h24m13.170141693s) [noise=true , address=77.90.31.14:4678     , piece_id=N2A4RJF5FFH3Y7J6EO4IUJ6HHVJBODEIRKMNUQIXHLTXUAHOMDCA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.388158077 (522h24m13.259241123s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.480680246 (522h24m13.351763281s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.691531151 (522h24m13.562614196s) [noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.976152859 (522h24m13.847235904s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.980566223 (522h24m13.851649258s) [noise=true , address=145.79.227.30:4777   , piece_id=FZ55XXJO5UC4SYAOX5PBHKD2FQG43WDEJDQGAWTMAYVJ2XQWBKMQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054854099 (522h24m13.925937134s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054871502 (522h24m13.925954537s) [seg_pos={0 0}, upload=458284]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:53:14.055044472 (522h24m13.926127507s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055051375 (522h24m13.92613441s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055334113 (522h24m13.926417148s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection
Dec 27 09:53:14.055340785 (522h24m13.926423831s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055487816 (522h24m13.926570851s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055494018 (522h24m13.926577053s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055610760 (522h24m13.926693795s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055616471 (522h24m13.926699516s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055630578 (522h24m13.926713613s) [seg_pos={0 0}, upload=458284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:53:14.055757609 (522h24m13.926840644s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:14.055762910 (522h24m13.926845945s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:14.343772328 (522h24m14.214855373s) [upload=458284]: Done uploading segments...
Dec 27 09:53:14.343848974 (522h24m14.214932009s) [upload=458284]: Upload finished. err: <nil>
Duration: 36.727755806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [.......X........X.............X........................X.X..]
duration: 
53.68598321s
trace: 
https://jaeger.dev.storj.io/trace/d450b6ce193c728
FAIL (2025-12-27 09:56:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233076300 (522h24m11.104159345s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233549011 (522h24m11.104632056s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233994970 (522h24m11.105078025s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234029806 (522h24m11.105112842s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234387447 (522h24m11.105470482s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234549115 (522h24m11.10563215s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234743615 (522h24m11.10582666s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235123268 (522h24m11.106206303s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235636846 (522h24m11.106719891s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235999697 (522h24m11.107082772s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236444434 (522h24m11.107527469s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236878911 (522h24m11.107961956s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237221313 (522h24m11.108304348s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237993243 (522h24m11.109076278s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.246722595 (522h24m11.11780568s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:12.530330220 (522h24m12.401413255s) [upload=458284]: Next returned nil segment
Dec 27 09:53:12.530349887 (522h24m12.401432922s) [upload=458284]: Waiting for error group managing segments...
Dec 27 09:53:12.876532286 (522h24m12.747615331s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:12.965370209 (522h24m12.836453244s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.115211782 (522h24m12.986294817s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.140777494 (522h24m13.011860529s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.169081760 (522h24m13.040164805s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.280115036 (522h24m13.151198071s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.293944019 (522h24m13.165027094s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.299058658 (522h24m13.170141693s) [noise=true , address=77.90.31.14:4678     , piece_id=N2A4RJF5FFH3Y7J6EO4IUJ6HHVJBODEIRKMNUQIXHLTXUAHOMDCA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.388158077 (522h24m13.259241123s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.480680246 (522h24m13.351763281s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.691531151 (522h24m13.562614196s) [noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.976152859 (522h24m13.847235904s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.980566223 (522h24m13.851649258s) [noise=true , address=145.79.227.30:4777   , piece_id=FZ55XXJO5UC4SYAOX5PBHKD2FQG43WDEJDQGAWTMAYVJ2XQWBKMQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054854099 (522h24m13.925937134s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054871502 (522h24m13.925954537s) [seg_pos={0 0}, upload=458284]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:53:14.055044472 (522h24m13.926127507s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055051375 (522h24m13.92613441s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055334113 (522h24m13.926417148s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection
Dec 27 09:53:14.055340785 (522h24m13.926423831s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055487816 (522h24m13.926570851s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055494018 (522h24m13.926577053s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055610760 (522h24m13.926693795s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055616471 (522h24m13.926699516s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055630578 (522h24m13.926713613s) [seg_pos={0 0}, upload=458284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:53:14.055757609 (522h24m13.926840644s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:14.055762910 (522h24m13.926845945s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:14.343772328 (522h24m14.214855373s) [upload=458284]: Done uploading segments...
Dec 27 09:53:14.343848974 (522h24m14.214932009s) [upload=458284]: Upload finished. err: <nil>
Duration: 53.686000383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [.X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX..]
duration: 
5.866119606s
trace: 
https://jaeger.dev.storj.io/trace/4b1782156d02b8ab
FAIL (2025-12-27 09:55:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ) context deadline exceeded
======LOGS======
Dec 27 09:54:08.161114044 (522h25m8.032197089s) [upload=458299]: Starting upload
Dec 27 09:54:08.161204756 (522h25m8.032287811s) [upload=458299]: Done starting upload
Dec 27 09:54:08.161355674 (522h25m8.032438719s) [upload=458299]: Uploading segments...
Dec 27 09:54:08.161461686 (522h25m8.032544731s) [upload=458299]: Got next segment. Inline: false
Dec 27 09:54:08.161504096 (522h25m8.032587131s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:54:08.161510388 (522h25m8.032593433s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:54:08.405295736 (522h25m8.276378782s) [seg_pos={0 0}, upload=458299]: Begin upload segment...
Dec 27 09:54:08.405355591 (522h25m8.276438636s) [seg_pos={0 0}, upload=458299]: Done begin upload segment.
Dec 27 09:54:08.405428519 (522h25m8.276511565s) [noise=true , address=77.90.31.10:4519     , piece_id=RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.405446394 (522h25m8.276529429s) [noise=true , address=45.91.116.11:4119    , piece_id=UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406267978 (522h25m8.277351023s) [noise=true , address=145.79.227.42:4122   , piece_id=R2ZBRDBBWA6A64CZCDKVTJZUD5GR73GBERFIPQGIKK4B67ETTLJQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406429025 (522h25m8.27751207s) [noise=true , address=145.79.227.52:4446   , piece_id=QWFSZD4S27SZIVBG6GF2AUNNXIQ5FEXQJNOZHG6GU5IEQOWW4BRQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406995994 (522h25m8.278079039s) [noise=true , address=145.79.227.27:4690   , piece_id=TLZCCT6XWUMGZ75RRWAJVIWX7LQ5TGGZZ7V4MRTWAIZTSJUKB5EQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.407478413 (522h25m8.278561458s) [noise=true , address=145.79.227.23:4546   , piece_id=VVK7SS5JKZ5LUI7BOY5MNVERHCP76FZ6C7QE6FKVQP7PTAOSRO4A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.407630302 (522h25m8.278713347s) [noise=true , address=145.79.227.22:4509   , piece_id=QF6OMEYUI4HBMCODNEG6HBWAQEB66JKI6OULHU2TU4PREOOEU7MA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.408420797 (522h25m8.279503842s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.408749352 (522h25m8.279832388s) [noise=true , address=145.79.227.36:4517   , piece_id=UPCE6SLFDHRWPJADJ64AZN3Z6TVKXX4M6PBHBLLOF5ROUYPYLV5A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.409332804 (522h25m8.280415849s) [noise=true , address=145.79.227.11:4729   , piece_id=MNAU5AE34IF74JBD6OBJCPXC46TVSCBSXUF72OS3LWPIRHXWKJHA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.410037105 (522h25m8.28112015s) [noise=true , address=145.79.227.3:4149    , piece_id=KZHUIOGQYOIZDL2TXI3ZI6NEDO6Z4DSKQ23IDDXQ5XZ4VL3THYTQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.410550873 (522h25m8.281633918s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.411127892 (522h25m8.282210937s) [noise=true , address=145.79.227.48:4303   , piece_id=KOS4TUAP5RFTKSTWPI4NFQGAVPSSGVVIUGXYGIIXTNBR26AUE3YA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.411893139 (522h25m8.282976185s) [noise=true , address=145.79.227.7:4295    , piece_id=HOQOY3C44QEZNJ57FHMBK7RCLVEONSN6O3OAQKBIFCHKEQR7CJTA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.414494162 (522h25m8.285577197s) [noise=true , address=45.91.116.10:4082    , piece_id=74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.414916947 (522h25m8.285999983s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:10.129294840 (522h25m10.000377886s) [upload=458299]: Next returned nil segment
Dec 27 09:54:10.129309929 (522h25m10.000392974s) [upload=458299]: Waiting for error group managing segments...
Dec 27 09:54:10.413951996 (522h25m10.285035031s) [noise=true , address=145.79.227.3:4149    , piece_id=KZHUIOGQYOIZDL2TXI3ZI6NEDO6Z4DSKQ23IDDXQ5XZ4VL3THYTQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.415467491 (522h25m10.286550536s) [noise=true , address=145.79.227.23:4546   , piece_id=VVK7SS5JKZ5LUI7BOY5MNVERHCP76FZ6C7QE6FKVQP7PTAOSRO4A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.417547291 (522h25m10.288630326s) [noise=true , address=145.79.227.27:4690   , piece_id=TLZCCT6XWUMGZ75RRWAJVIWX7LQ5TGGZZ7V4MRTWAIZTSJUKB5EQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.421607152 (522h25m10.292690187s) [noise=true , address=145.79.227.42:4122   , piece_id=R2ZBRDBBWA6A64CZCDKVTJZUD5GR73GBERFIPQGIKK4B67ETTLJQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.422320620 (522h25m10.293403655s) [noise=true , address=145.79.227.48:4303   , piece_id=KOS4TUAP5RFTKSTWPI4NFQGAVPSSGVVIUGXYGIIXTNBR26AUE3YA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.424943936 (522h25m10.296026971s) [noise=true , address=145.79.227.22:4509   , piece_id=QF6OMEYUI4HBMCODNEG6HBWAQEB66JKI6OULHU2TU4PREOOEU7MA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.429015790 (522h25m10.300098835s) [noise=true , address=145.79.227.11:4729   , piece_id=MNAU5AE34IF74JBD6OBJCPXC46TVSCBSXUF72OS3LWPIRHXWKJHA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.430957166 (522h25m10.302040211s) [noise=true , address=145.79.227.52:4446   , piece_id=QWFSZD4S27SZIVBG6GF2AUNNXIQ5FEXQJNOZHG6GU5IEQOWW4BRQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.435547897 (522h25m10.306630943s) [noise=true , address=145.79.227.7:4295    , piece_id=HOQOY3C44QEZNJ57FHMBK7RCLVEONSN6O3OAQKBIFCHKEQR7CJTA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.441473691 (522h25m10.312556726s) [noise=true , address=145.79.227.36:4517   , piece_id=UPCE6SLFDHRWPJADJ64AZN3Z6TVKXX4M6PBHBLLOF5ROUYPYLV5A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.674996988 (522h25m10.546080033s) [noise=true , address=45.91.116.10:4082    , piece_id=74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.678958151 (522h25m10.550041196s) [noise=true , address=45.91.116.11:4119    , piece_id=UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:11.339111945 (522h25m11.21019498s) [noise=true , address=77.90.31.10:4519     , piece_id=RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:11.339132645 (522h25m11.21021568s) [seg_pos={0 0}, upload=458299]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:54:11.339350700 (522h25m11.210433735s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 09:54:11.339357833 (522h25m11.210440868s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.339678764 (522h25m11.210761809s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 09:54:11.339688473 (522h25m11.210771518s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.339980158 (522h25m11.211063193s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 27 09:54:11.340003012 (522h25m11.211086057s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.340040964 (522h25m11.211124009s) [seg_pos={0 0}, upload=458299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:54:11.340165331 (522h25m11.211248376s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:54:11.340173836 (522h25m11.211256882s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:54:11.632713462 (522h25m11.503796507s) [upload=458299]: Done uploading segments...
Dec 27 09:54:11.632736386 (522h25m11.503819421s) [upload=458299]: Upload finished. err: <nil>
Duration: 5.866152066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [........X........X.............X........................X.X.]
duration: 
53.758874529s
trace: 
https://jaeger.dev.storj.io/trace/d450b6ce193c728
FAIL (2025-12-27 09:55:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233076300 (522h24m11.104159345s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233549011 (522h24m11.104632056s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233994970 (522h24m11.105078025s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234029806 (522h24m11.105112842s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234387447 (522h24m11.105470482s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234549115 (522h24m11.10563215s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234743615 (522h24m11.10582666s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235123268 (522h24m11.106206303s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235636846 (522h24m11.106719891s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235999697 (522h24m11.107082772s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236444434 (522h24m11.107527469s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236878911 (522h24m11.107961956s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237221313 (522h24m11.108304348s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237993243 (522h24m11.109076278s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.246722595 (522h24m11.11780568s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:12.530330220 (522h24m12.401413255s) [upload=458284]: Next returned nil segment
Dec 27 09:53:12.530349887 (522h24m12.401432922s) [upload=458284]: Waiting for error group managing segments...
Dec 27 09:53:12.876532286 (522h24m12.747615331s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:12.965370209 (522h24m12.836453244s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.115211782 (522h24m12.986294817s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.140777494 (522h24m13.011860529s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.169081760 (522h24m13.040164805s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.280115036 (522h24m13.151198071s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.293944019 (522h24m13.165027094s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.299058658 (522h24m13.170141693s) [noise=true , address=77.90.31.14:4678     , piece_id=N2A4RJF5FFH3Y7J6EO4IUJ6HHVJBODEIRKMNUQIXHLTXUAHOMDCA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.388158077 (522h24m13.259241123s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.480680246 (522h24m13.351763281s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.691531151 (522h24m13.562614196s) [noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.976152859 (522h24m13.847235904s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.980566223 (522h24m13.851649258s) [noise=true , address=145.79.227.30:4777   , piece_id=FZ55XXJO5UC4SYAOX5PBHKD2FQG43WDEJDQGAWTMAYVJ2XQWBKMQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054854099 (522h24m13.925937134s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054871502 (522h24m13.925954537s) [seg_pos={0 0}, upload=458284]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:53:14.055044472 (522h24m13.926127507s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055051375 (522h24m13.92613441s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055334113 (522h24m13.926417148s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection
Dec 27 09:53:14.055340785 (522h24m13.926423831s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055487816 (522h24m13.926570851s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055494018 (522h24m13.926577053s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055610760 (522h24m13.926693795s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055616471 (522h24m13.926699516s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055630578 (522h24m13.926713613s) [seg_pos={0 0}, upload=458284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:53:14.055757609 (522h24m13.926840644s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:14.055762910 (522h24m13.926845945s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:14.343772328 (522h24m14.214855373s) [upload=458284]: Done uploading segments...
Dec 27 09:53:14.343848974 (522h24m14.214932009s) [upload=458284]: Upload finished. err: <nil>
Duration: 53.758904175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [..X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX.]
duration: 
5.687411985s
trace: 
https://jaeger.dev.storj.io/trace/4b1782156d02b8ab
FAIL (2025-12-27 09:55:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ) context deadline exceeded
======LOGS======
Dec 27 09:54:08.161114044 (522h25m8.032197089s) [upload=458299]: Starting upload
Dec 27 09:54:08.161204756 (522h25m8.032287811s) [upload=458299]: Done starting upload
Dec 27 09:54:08.161355674 (522h25m8.032438719s) [upload=458299]: Uploading segments...
Dec 27 09:54:08.161461686 (522h25m8.032544731s) [upload=458299]: Got next segment. Inline: false
Dec 27 09:54:08.161504096 (522h25m8.032587131s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:54:08.161510388 (522h25m8.032593433s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:54:08.405295736 (522h25m8.276378782s) [seg_pos={0 0}, upload=458299]: Begin upload segment...
Dec 27 09:54:08.405355591 (522h25m8.276438636s) [seg_pos={0 0}, upload=458299]: Done begin upload segment.
Dec 27 09:54:08.405428519 (522h25m8.276511565s) [noise=true , address=77.90.31.10:4519     , piece_id=RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.405446394 (522h25m8.276529429s) [noise=true , address=45.91.116.11:4119    , piece_id=UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406267978 (522h25m8.277351023s) [noise=true , address=145.79.227.42:4122   , piece_id=R2ZBRDBBWA6A64CZCDKVTJZUD5GR73GBERFIPQGIKK4B67ETTLJQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406429025 (522h25m8.27751207s) [noise=true , address=145.79.227.52:4446   , piece_id=QWFSZD4S27SZIVBG6GF2AUNNXIQ5FEXQJNOZHG6GU5IEQOWW4BRQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406995994 (522h25m8.278079039s) [noise=true , address=145.79.227.27:4690   , piece_id=TLZCCT6XWUMGZ75RRWAJVIWX7LQ5TGGZZ7V4MRTWAIZTSJUKB5EQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.407478413 (522h25m8.278561458s) [noise=true , address=145.79.227.23:4546   , piece_id=VVK7SS5JKZ5LUI7BOY5MNVERHCP76FZ6C7QE6FKVQP7PTAOSRO4A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.407630302 (522h25m8.278713347s) [noise=true , address=145.79.227.22:4509   , piece_id=QF6OMEYUI4HBMCODNEG6HBWAQEB66JKI6OULHU2TU4PREOOEU7MA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.408420797 (522h25m8.279503842s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.408749352 (522h25m8.279832388s) [noise=true , address=145.79.227.36:4517   , piece_id=UPCE6SLFDHRWPJADJ64AZN3Z6TVKXX4M6PBHBLLOF5ROUYPYLV5A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.409332804 (522h25m8.280415849s) [noise=true , address=145.79.227.11:4729   , piece_id=MNAU5AE34IF74JBD6OBJCPXC46TVSCBSXUF72OS3LWPIRHXWKJHA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.410037105 (522h25m8.28112015s) [noise=true , address=145.79.227.3:4149    , piece_id=KZHUIOGQYOIZDL2TXI3ZI6NEDO6Z4DSKQ23IDDXQ5XZ4VL3THYTQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.410550873 (522h25m8.281633918s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.411127892 (522h25m8.282210937s) [noise=true , address=145.79.227.48:4303   , piece_id=KOS4TUAP5RFTKSTWPI4NFQGAVPSSGVVIUGXYGIIXTNBR26AUE3YA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.411893139 (522h25m8.282976185s) [noise=true , address=145.79.227.7:4295    , piece_id=HOQOY3C44QEZNJ57FHMBK7RCLVEONSN6O3OAQKBIFCHKEQR7CJTA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.414494162 (522h25m8.285577197s) [noise=true , address=45.91.116.10:4082    , piece_id=74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.414916947 (522h25m8.285999983s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:10.129294840 (522h25m10.000377886s) [upload=458299]: Next returned nil segment
Dec 27 09:54:10.129309929 (522h25m10.000392974s) [upload=458299]: Waiting for error group managing segments...
Dec 27 09:54:10.413951996 (522h25m10.285035031s) [noise=true , address=145.79.227.3:4149    , piece_id=KZHUIOGQYOIZDL2TXI3ZI6NEDO6Z4DSKQ23IDDXQ5XZ4VL3THYTQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.415467491 (522h25m10.286550536s) [noise=true , address=145.79.227.23:4546   , piece_id=VVK7SS5JKZ5LUI7BOY5MNVERHCP76FZ6C7QE6FKVQP7PTAOSRO4A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.417547291 (522h25m10.288630326s) [noise=true , address=145.79.227.27:4690   , piece_id=TLZCCT6XWUMGZ75RRWAJVIWX7LQ5TGGZZ7V4MRTWAIZTSJUKB5EQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.421607152 (522h25m10.292690187s) [noise=true , address=145.79.227.42:4122   , piece_id=R2ZBRDBBWA6A64CZCDKVTJZUD5GR73GBERFIPQGIKK4B67ETTLJQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.422320620 (522h25m10.293403655s) [noise=true , address=145.79.227.48:4303   , piece_id=KOS4TUAP5RFTKSTWPI4NFQGAVPSSGVVIUGXYGIIXTNBR26AUE3YA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.424943936 (522h25m10.296026971s) [noise=true , address=145.79.227.22:4509   , piece_id=QF6OMEYUI4HBMCODNEG6HBWAQEB66JKI6OULHU2TU4PREOOEU7MA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.429015790 (522h25m10.300098835s) [noise=true , address=145.79.227.11:4729   , piece_id=MNAU5AE34IF74JBD6OBJCPXC46TVSCBSXUF72OS3LWPIRHXWKJHA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.430957166 (522h25m10.302040211s) [noise=true , address=145.79.227.52:4446   , piece_id=QWFSZD4S27SZIVBG6GF2AUNNXIQ5FEXQJNOZHG6GU5IEQOWW4BRQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.435547897 (522h25m10.306630943s) [noise=true , address=145.79.227.7:4295    , piece_id=HOQOY3C44QEZNJ57FHMBK7RCLVEONSN6O3OAQKBIFCHKEQR7CJTA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.441473691 (522h25m10.312556726s) [noise=true , address=145.79.227.36:4517   , piece_id=UPCE6SLFDHRWPJADJ64AZN3Z6TVKXX4M6PBHBLLOF5ROUYPYLV5A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.674996988 (522h25m10.546080033s) [noise=true , address=45.91.116.10:4082    , piece_id=74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.678958151 (522h25m10.550041196s) [noise=true , address=45.91.116.11:4119    , piece_id=UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:11.339111945 (522h25m11.21019498s) [noise=true , address=77.90.31.10:4519     , piece_id=RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:11.339132645 (522h25m11.21021568s) [seg_pos={0 0}, upload=458299]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:54:11.339350700 (522h25m11.210433735s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 09:54:11.339357833 (522h25m11.210440868s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.339678764 (522h25m11.210761809s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 09:54:11.339688473 (522h25m11.210771518s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.339980158 (522h25m11.211063193s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 27 09:54:11.340003012 (522h25m11.211086057s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.340040964 (522h25m11.211124009s) [seg_pos={0 0}, upload=458299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:54:11.340165331 (522h25m11.211248376s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:54:11.340173836 (522h25m11.211256882s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:54:11.632713462 (522h25m11.503796507s) [upload=458299]: Done uploading segments...
Dec 27 09:54:11.632736386 (522h25m11.503819421s) [upload=458299]: Upload finished. err: <nil>
Duration: 5.687442212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....XX]
duration: 
30.003191038s
trace: 
https://jaeger.dev.storj.io/trace/4b1782156d02b8ab
FAIL (2025-12-27 09:54:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ) context deadline exceeded
======LOGS======
Dec 27 09:54:08.161114044 (522h25m8.032197089s) [upload=458299]: Starting upload
Dec 27 09:54:08.161204756 (522h25m8.032287811s) [upload=458299]: Done starting upload
Dec 27 09:54:08.161355674 (522h25m8.032438719s) [upload=458299]: Uploading segments...
Dec 27 09:54:08.161461686 (522h25m8.032544731s) [upload=458299]: Got next segment. Inline: false
Dec 27 09:54:08.161504096 (522h25m8.032587131s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:54:08.161510388 (522h25m8.032593433s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:54:08.405295736 (522h25m8.276378782s) [seg_pos={0 0}, upload=458299]: Begin upload segment...
Dec 27 09:54:08.405355591 (522h25m8.276438636s) [seg_pos={0 0}, upload=458299]: Done begin upload segment.
Dec 27 09:54:08.405428519 (522h25m8.276511565s) [noise=true , address=77.90.31.10:4519     , piece_id=RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.405446394 (522h25m8.276529429s) [noise=true , address=45.91.116.11:4119    , piece_id=UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406267978 (522h25m8.277351023s) [noise=true , address=145.79.227.42:4122   , piece_id=R2ZBRDBBWA6A64CZCDKVTJZUD5GR73GBERFIPQGIKK4B67ETTLJQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406429025 (522h25m8.27751207s) [noise=true , address=145.79.227.52:4446   , piece_id=QWFSZD4S27SZIVBG6GF2AUNNXIQ5FEXQJNOZHG6GU5IEQOWW4BRQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.406995994 (522h25m8.278079039s) [noise=true , address=145.79.227.27:4690   , piece_id=TLZCCT6XWUMGZ75RRWAJVIWX7LQ5TGGZZ7V4MRTWAIZTSJUKB5EQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.407478413 (522h25m8.278561458s) [noise=true , address=145.79.227.23:4546   , piece_id=VVK7SS5JKZ5LUI7BOY5MNVERHCP76FZ6C7QE6FKVQP7PTAOSRO4A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.407630302 (522h25m8.278713347s) [noise=true , address=145.79.227.22:4509   , piece_id=QF6OMEYUI4HBMCODNEG6HBWAQEB66JKI6OULHU2TU4PREOOEU7MA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.408420797 (522h25m8.279503842s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.408749352 (522h25m8.279832388s) [noise=true , address=145.79.227.36:4517   , piece_id=UPCE6SLFDHRWPJADJ64AZN3Z6TVKXX4M6PBHBLLOF5ROUYPYLV5A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.409332804 (522h25m8.280415849s) [noise=true , address=145.79.227.11:4729   , piece_id=MNAU5AE34IF74JBD6OBJCPXC46TVSCBSXUF72OS3LWPIRHXWKJHA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.410037105 (522h25m8.28112015s) [noise=true , address=145.79.227.3:4149    , piece_id=KZHUIOGQYOIZDL2TXI3ZI6NEDO6Z4DSKQ23IDDXQ5XZ4VL3THYTQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.410550873 (522h25m8.281633918s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.411127892 (522h25m8.282210937s) [noise=true , address=145.79.227.48:4303   , piece_id=KOS4TUAP5RFTKSTWPI4NFQGAVPSSGVVIUGXYGIIXTNBR26AUE3YA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.411893139 (522h25m8.282976185s) [noise=true , address=145.79.227.7:4295    , piece_id=HOQOY3C44QEZNJ57FHMBK7RCLVEONSN6O3OAQKBIFCHKEQR7CJTA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.414494162 (522h25m8.285577197s) [noise=true , address=45.91.116.10:4082    , piece_id=74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:08.414916947 (522h25m8.285999983s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Uploading piece...
Dec 27 09:54:10.129294840 (522h25m10.000377886s) [upload=458299]: Next returned nil segment
Dec 27 09:54:10.129309929 (522h25m10.000392974s) [upload=458299]: Waiting for error group managing segments...
Dec 27 09:54:10.413951996 (522h25m10.285035031s) [noise=true , address=145.79.227.3:4149    , piece_id=KZHUIOGQYOIZDL2TXI3ZI6NEDO6Z4DSKQ23IDDXQ5XZ4VL3THYTQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.415467491 (522h25m10.286550536s) [noise=true , address=145.79.227.23:4546   , piece_id=VVK7SS5JKZ5LUI7BOY5MNVERHCP76FZ6C7QE6FKVQP7PTAOSRO4A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.417547291 (522h25m10.288630326s) [noise=true , address=145.79.227.27:4690   , piece_id=TLZCCT6XWUMGZ75RRWAJVIWX7LQ5TGGZZ7V4MRTWAIZTSJUKB5EQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.421607152 (522h25m10.292690187s) [noise=true , address=145.79.227.42:4122   , piece_id=R2ZBRDBBWA6A64CZCDKVTJZUD5GR73GBERFIPQGIKK4B67ETTLJQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.422320620 (522h25m10.293403655s) [noise=true , address=145.79.227.48:4303   , piece_id=KOS4TUAP5RFTKSTWPI4NFQGAVPSSGVVIUGXYGIIXTNBR26AUE3YA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.424943936 (522h25m10.296026971s) [noise=true , address=145.79.227.22:4509   , piece_id=QF6OMEYUI4HBMCODNEG6HBWAQEB66JKI6OULHU2TU4PREOOEU7MA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.429015790 (522h25m10.300098835s) [noise=true , address=145.79.227.11:4729   , piece_id=MNAU5AE34IF74JBD6OBJCPXC46TVSCBSXUF72OS3LWPIRHXWKJHA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.430957166 (522h25m10.302040211s) [noise=true , address=145.79.227.52:4446   , piece_id=QWFSZD4S27SZIVBG6GF2AUNNXIQ5FEXQJNOZHG6GU5IEQOWW4BRQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.435547897 (522h25m10.306630943s) [noise=true , address=145.79.227.7:4295    , piece_id=HOQOY3C44QEZNJ57FHMBK7RCLVEONSN6O3OAQKBIFCHKEQR7CJTA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.441473691 (522h25m10.312556726s) [noise=true , address=145.79.227.36:4517   , piece_id=UPCE6SLFDHRWPJADJ64AZN3Z6TVKXX4M6PBHBLLOF5ROUYPYLV5A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.674996988 (522h25m10.546080033s) [noise=true , address=45.91.116.10:4082    , piece_id=74PBLVW5QPV3JE32MLBPVEL6QY6J4B23Q4WIH5ACJ6C7OVM5XOIA, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:10.678958151 (522h25m10.550041196s) [noise=true , address=45.91.116.11:4119    , piece_id=UTO5S7J75JSSZA3TEPUCZSROKRZLUGPYTNWI2YH5YT3DCBRDKB3Q, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:11.339111945 (522h25m11.21019498s) [noise=true , address=77.90.31.10:4519     , piece_id=RC6GV573V72SBRT5OKVC7YWVEUGI5PGJY6OMQ2N5IQGFN5OUD2MQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: <nil>
Dec 27 09:54:11.339132645 (522h25m11.21021568s) [seg_pos={0 0}, upload=458299]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:54:11.339350700 (522h25m11.210433735s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 09:54:11.339357833 (522h25m11.210440868s) [noise=true , address=77.90.31.21:4880     , piece_id=NIGL4P225WDXGVRSSPNWGUVDFLNIH3GRIDANZJYLRGOJ64RSAE4A, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.339678764 (522h25m11.210761809s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 09:54:11.339688473 (522h25m11.210771518s) [noise=true , address=45.91.116.8:4048     , piece_id=QXOCI2Y5475YIYYH2IBFJT3XYGKPMRLSRAJ55I6QLCPQSSS2V6AQ, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.339980158 (522h25m11.211063193s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 27 09:54:11.340003012 (522h25m11.211086057s) [noise=true , address=45.91.116.5:4297     , piece_id=QVXJHCE5D2CHR4MGN2PGD4QSBH266EKO2PUCOQBJU4AI5ZHSXBMQ, seg_pos={0 0}, upload=458299]: Long tail cancellation due to optimal threshold
Dec 27 09:54:11.340040964 (522h25m11.211124009s) [seg_pos={0 0}, upload=458299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:54:11.340165331 (522h25m11.211248376s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:54:11.340173836 (522h25m11.211256882s) [upload=458299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:54:11.632713462 (522h25m11.503796507s) [upload=458299]: Done uploading segments...
Dec 27 09:54:11.632736386 (522h25m11.503819421s) [upload=458299]: Upload finished. err: <nil>
Duration: 30.003215635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [.........X........X.............X........................X.X]
duration: 
1m0.001703849s
trace: 
https://jaeger.dev.storj.io/trace/d450b6ce193c728
FAIL (2025-12-27 09:54:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233076300 (522h24m11.104159345s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233549011 (522h24m11.104632056s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.233994970 (522h24m11.105078025s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234029806 (522h24m11.105112842s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234387447 (522h24m11.105470482s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234549115 (522h24m11.10563215s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.234743615 (522h24m11.10582666s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235123268 (522h24m11.106206303s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235636846 (522h24m11.106719891s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.235999697 (522h24m11.107082772s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236444434 (522h24m11.107527469s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.236878911 (522h24m11.107961956s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237221313 (522h24m11.108304348s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.237993243 (522h24m11.109076278s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:11.246722595 (522h24m11.11780568s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Uploading piece...
Dec 27 09:53:12.530330220 (522h24m12.401413255s) [upload=458284]: Next returned nil segment
Dec 27 09:53:12.530349887 (522h24m12.401432922s) [upload=458284]: Waiting for error group managing segments...
Dec 27 09:53:12.876532286 (522h24m12.747615331s) [noise=true , address=64.49.13.27:4968     , piece_id=ZE6MFN5JZPJA6J5YQZBEEALJSIVKE6RSD6VJMDR6EUTE2SUNSNSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:12.965370209 (522h24m12.836453244s) [noise=true , address=64.49.13.140:4516    , piece_id=M72M7NDJK2D4IKENHY6SHUSTYGHFS4F4MF3WCL6EB4BWUNXZEDSQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.115211782 (522h24m12.986294817s) [noise=true , address=145.79.227.34:4933   , piece_id=3VKBMT63JKGVMHKSPQ44Y5WRUFPWTLRLT44OVLLG752A37KHOD3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.140777494 (522h24m13.011860529s) [noise=true , address=145.79.227.40:4053   , piece_id=N35KD36XFCCBYCWSBSFYYJOGFBRDGUHLX4ZQXBKAUYAZXBO7KLQQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.169081760 (522h24m13.040164805s) [noise=true , address=193.149.164.143:4701 , piece_id=QE6GENP455TZ47A63RKNWMAQXZREIYX54P2XISROXKLNX5XJLP7A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.280115036 (522h24m13.151198071s) [noise=true , address=77.90.31.15:4715     , piece_id=IUDCW3IZVVE6TWJ4QLPGP453NUY622WVI7YKIX5X64PN25Q2AIUQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.293944019 (522h24m13.165027094s) [noise=true , address=77.90.31.13:4631     , piece_id=UCW7Q3YHR7SNBNM5526WHK5QMVP62CR4IN3FKWM44J2QQVYABCXA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.299058658 (522h24m13.170141693s) [noise=true , address=77.90.31.14:4678     , piece_id=N2A4RJF5FFH3Y7J6EO4IUJ6HHVJBODEIRKMNUQIXHLTXUAHOMDCA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.388158077 (522h24m13.259241123s) [noise=true , address=77.90.31.12:4614     , piece_id=4PVEPPL2WANSAFX5U6ZFUD3VXUHKWEHFHPBKVRQUASLHYFZDQMZA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.480680246 (522h24m13.351763281s) [noise=true , address=77.90.31.19:4871     , piece_id=PMJPC4YSBWK2CRLRUW4UNNBFUK4EMWSXPGYSPXCCNIU2HN75PUBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.691531151 (522h24m13.562614196s) [noise=true , address=45.91.116.8:4066     , piece_id=7QDTEU2BZ7OJMUTLICUAZXE4KEPO4GOIJGAQF7YSFOO667BOEZ3A, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.976152859 (522h24m13.847235904s) [noise=true , address=77.90.31.17:4774     , piece_id=YXX4VGBQDKXAQZSEVVOXEXORH573JWXEM2GRKQ5LPHMWQYWP3B3Q, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:13.980566223 (522h24m13.851649258s) [noise=true , address=145.79.227.30:4777   , piece_id=FZ55XXJO5UC4SYAOX5PBHKD2FQG43WDEJDQGAWTMAYVJ2XQWBKMQ, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054854099 (522h24m13.925937134s) [noise=true , address=77.90.31.10:4542     , piece_id=6VM6HFUQABNFD76PZT2BF67B5OV4VK5IILZHDJH3NRKN4SG4STBA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: <nil>
Dec 27 09:53:14.054871502 (522h24m13.925954537s) [seg_pos={0 0}, upload=458284]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:53:14.055044472 (522h24m13.926127507s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055051375 (522h24m13.92613441s) [noise=true , address=45.91.116.10:4089    , piece_id=IMXU2F726QKXPISV4SV4HOYUWVEYBGDJS67X3SSQIOO7DOIFGCHA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055334113 (522h24m13.926417148s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45566->64.49.13.169:4709: use of closed network connection
Dec 27 09:53:14.055340785 (522h24m13.926423831s) [noise=true , address=64.49.13.169:4709    , piece_id=2PZRM5JQKALUX4BM4P4IYKYFYUNGUPYWNAFLFKEPGMWVVARNFTYA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055487816 (522h24m13.926570851s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055494018 (522h24m13.926577053s) [noise=true , address=77.90.31.18:4839     , piece_id=7JSANUKASHZOTLE2TBK22SSHNPTCIQLYGCTYIB6L2NIPGXZ5RMKA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055610760 (522h24m13.926693795s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF
Dec 27 09:53:14.055616471 (522h24m13.926699516s) [noise=true , address=145.79.227.44:4218   , piece_id=YQ5JABHXAX5ZS7Y4OL7PXW6ZWHBVVBAVAKQ4VB3L7JFYBEAU47QA, seg_pos={0 0}, upload=458284]: Long tail cancellation due to optimal threshold
Dec 27 09:53:14.055630578 (522h24m13.926713613s) [seg_pos={0 0}, upload=458284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:53:14.055757609 (522h24m13.926840644s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:14.055762910 (522h24m13.926845945s) [upload=458284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:14.343772328 (522h24m14.214855373s) [upload=458284]: Done uploading segments...
Dec 27 09:53:14.343848974 (522h24m14.214932009s) [upload=458284]: Upload finished. err: <nil>
Duration: 1m0.001727074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [X...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....X]
duration: 
30.002010309s
trace: 
https://jaeger.dev.storj.io/trace/26f4a29d38554bf2
FAIL (2025-12-27 09:53:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======

Dec 27 09:53:27.459904002 (522h24m27.330987047s) [upload=458290]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:53:27.459911126 (522h24m27.330994171s) [upload=458290]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:53:27.703681436 (522h24m27.574764481s) [seg_pos={0 0}, upload=458290]: Begin upload segment...
Dec 27 09:53:27.703737682 (522h24m27.574820727s) [seg_pos={0 0}, upload=458290]: Done begin upload segment.
Dec 27 09:53:27.703766046 (522h24m27.574849091s) [noise=true , address=45.91.116.5:4300     , piece_id=4FOKHTY6KELLIRPV2MHFBH5FSKXFRYPJHA7YNHGKF4A5PKFNFSTQ, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.703779892 (522h24m27.574862937s) [noise=true , address=145.79.227.25:4617   , piece_id=V3UJZZCHHFPJUS2Y7RUQA4BHOLSGBWKNBP4MQMWG2IURRYJUKXMQ, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.703872299 (522h24m27.574955334s) [noise=true , address=145.79.227.39:4014   , piece_id=6EEH4URFGW2XAQQUWZUV6R4QZGPK7O7HMDNV6MVV2JMNLHJDGCKQ, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.704307578 (522h24m27.575390613s) [noise=true , address=145.79.227.35:4482   , piece_id=DCSQGU5ECPZ5LWAPEAMT67QFWZ7N4HR2CVGE4DWGFRDF7Y7DBXTA, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.704685828 (522h24m27.575768863s) [noise=true , address=145.79.227.17:4972   , piece_id=QABWHKF66YJOGHKEEWJAE7BA4CKYFUA7X4XEFMZ62T3QW34FWNUA, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.705064178 (522h24m27.576147223s) [noise=true , address=145.79.227.46:4603   , piece_id=GKUQRFAXHRLWFYLJ22VNISO564UMOTKNOQJYYEYDONG74EJVH5LA, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.705432049 (522h24m27.576515094s) [noise=true , address=145.79.227.28:4725   , piece_id=4TA4JW4KROM4BH4HUVCZWWLU7OYXFCFDR6H3ZKAGPZS5O6UQNIWQ, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.706032813 (522h24m27.577115848s) [noise=true , address=77.90.31.19:4841     , piece_id=DDZYZH5XGKY4E4SRASJMSPCVJ6DSAGMCXKGRSIECG7OEIQ5WFE5Q, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.706542864 (522h24m27.577625909s) [noise=true , address=45.91.116.8:4045     , piece_id=Z4IPCQZCSJQEFBLCSVFRD4RA2HNBRCVVPSO7TUCXI3KBX3UUZB7A, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.707020023 (522h24m27.578103068s) [noise=true , address=145.79.227.48:4301   , piece_id=22LUQ63CUUGPWNS4GRUOAFTENR3AMDEB377CGQSWIKLCN7FPRTJA, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.708897377 (522h24m27.579980422s) [noise=true , address=45.91.116.7:4012     , piece_id=CZ52CZKOJ25AE4RASL4GY2QF5QYDIRR44WWHGEOCBF7PITT4BCXQ, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.710730478 (522h24m27.581813523s) [noise=true , address=77.90.31.11:4554     , piece_id=VQJ7DW7QMMYHK5PIOGJ4GSQECIY4JJT3WJ44NK2NY5OESHBK6HWA, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.712017378 (522h24m27.583100423s) [noise=true , address=45.91.116.10:4084    , piece_id=3GOQGVRPIMN4B7URWQPDWLFC5VRA4IKTRGUZ6P4YKJA3M4HW5ERQ, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.713565437 (522h24m27.584648482s) [noise=true , address=145.79.227.51:4411   , piece_id=7N43U6CQRDQHX7G3JWS2INSARP3BIZH7UXGQRHO37ZRGDXTLGDGQ, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.715188497 (522h24m27.586271542s) [noise=true , address=145.79.227.42:4121   , piece_id=BEDNABVIPGOGRFKGA7U733J32ENDIJTMWVCQKUMFBGH4AYI3NBQA, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:27.716700015 (522h24m27.58778306s) [noise=true , address=45.91.116.11:4118    , piece_id=JGZ7UW72BIFHT3EL7I4C4GLDP7C7A2WVDQSXHWVOJCHN4IY6GACQ, seg_pos={0 0}, upload=458290]: Uploading piece...
Dec 27 09:53:29.324509223 (522h24m29.195592259s) [upload=458290]: Next returned nil segment
Dec 27 09:53:29.324522318 (522h24m29.195605353s) [upload=458290]: Waiting for error group managing segments...
Dec 27 09:53:29.710228749 (522h24m29.581311794s) [noise=true , address=145.79.227.48:4301   , piece_id=22LUQ63CUUGPWNS4GRUOAFTENR3AMDEB377CGQSWIKLCN7FPRTJA, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.715855552 (522h24m29.586938597s) [noise=true , address=145.79.227.46:4603   , piece_id=GKUQRFAXHRLWFYLJ22VNISO564UMOTKNOQJYYEYDONG74EJVH5LA, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.716330397 (522h24m29.587413442s) [noise=true , address=145.79.227.25:4617   , piece_id=V3UJZZCHHFPJUS2Y7RUQA4BHOLSGBWKNBP4MQMWG2IURRYJUKXMQ, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.732606539 (522h24m29.603689574s) [noise=true , address=145.79.227.39:4014   , piece_id=6EEH4URFGW2XAQQUWZUV6R4QZGPK7O7HMDNV6MVV2JMNLHJDGCKQ, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.735936470 (522h24m29.607019515s) [noise=true , address=145.79.227.51:4411   , piece_id=7N43U6CQRDQHX7G3JWS2INSARP3BIZH7UXGQRHO37ZRGDXTLGDGQ, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.739040941 (522h24m29.610123986s) [noise=true , address=145.79.227.17:4972   , piece_id=QABWHKF66YJOGHKEEWJAE7BA4CKYFUA7X4XEFMZ62T3QW34FWNUA, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.742692124 (522h24m29.613775159s) [noise=true , address=145.79.227.28:4725   , piece_id=4TA4JW4KROM4BH4HUVCZWWLU7OYXFCFDR6H3ZKAGPZS5O6UQNIWQ, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.744785089 (522h24m29.615868134s) [noise=true , address=145.79.227.42:4121   , piece_id=BEDNABVIPGOGRFKGA7U733J32ENDIJTMWVCQKUMFBGH4AYI3NBQA, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.802726760 (522h24m29.673809795s) [noise=true , address=145.79.227.35:4482   , piece_id=DCSQGU5ECPZ5LWAPEAMT67QFWZ7N4HR2CVGE4DWGFRDF7Y7DBXTA, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:29.994652810 (522h24m29.865735855s) [noise=true , address=77.90.31.11:4554     , piece_id=VQJ7DW7QMMYHK5PIOGJ4GSQECIY4JJT3WJ44NK2NY5OESHBK6HWA, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:30.015464999 (522h24m29.886548044s) [noise=true , address=77.90.31.19:4841     , piece_id=DDZYZH5XGKY4E4SRASJMSPCVJ6DSAGMCXKGRSIECG7OEIQ5WFE5Q, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:30.078063177 (522h24m29.949146212s) [noise=true , address=45.91.116.7:4012     , piece_id=CZ52CZKOJ25AE4RASL4GY2QF5QYDIRR44WWHGEOCBF7PITT4BCXQ, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:30.103668043 (522h24m29.974751088s) [noise=true , address=45.91.116.8:4045     , piece_id=Z4IPCQZCSJQEFBLCSVFRD4RA2HNBRCVVPSO7TUCXI3KBX3UUZB7A, seg_pos={0 0}, upload=458290]: Done uploading piece. err: <nil>
Dec 27 09:53:30.103687199 (522h24m29.974770234s) [seg_pos={0 0}, upload=458290]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:53:30.103962283 (522h24m29.975045318s) [noise=true , address=45.91.116.11:4118    , piece_id=JGZ7UW72BIFHT3EL7I4C4GLDP7C7A2WVDQSXHWVOJCHN4IY6GACQ, seg_pos={0 0}, upload=458290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:37138->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 09:53:30.103968985 (522h24m29.97505202s) [noise=true , address=45.91.116.11:4118    , piece_id=JGZ7UW72BIFHT3EL7I4C4GLDP7C7A2WVDQSXHWVOJCHN4IY6GACQ, seg_pos={0 0}, upload=458290]: Long tail cancellation due to optimal threshold
Dec 27 09:53:30.104224703 (522h24m29.975307738s) [noise=true , address=45.91.116.10:4084    , piece_id=3GOQGVRPIMN4B7URWQPDWLFC5VRA4IKTRGUZ6P4YKJA3M4HW5ERQ, seg_pos={0 0}, upload=458290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:39866->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:39866->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39866->45.91.116.10:4084: use of closed network connection
Dec 27 09:53:30.104232467 (522h24m29.975315512s) [noise=true , address=45.91.116.10:4084    , piece_id=3GOQGVRPIMN4B7URWQPDWLFC5VRA4IKTRGUZ6P4YKJA3M4HW5ERQ, seg_pos={0 0}, upload=458290]: Long tail cancellation due to optimal threshold
Dec 27 09:53:30.104314132 (522h24m29.975397177s) [noise=true , address=45.91.116.5:4300     , piece_id=4FOKHTY6KELLIRPV2MHFBH5FSKXFRYPJHA7YNHGKF4A5PKFNFSTQ, seg_pos={0 0}, upload=458290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:39352->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:39352->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39352->45.91.116.5:4300: use of closed network connection
Dec 27 09:53:30.104319814 (522h24m29.975402859s) [noise=true , address=45.91.116.5:4300     , piece_id=4FOKHTY6KELLIRPV2MHFBH5FSKXFRYPJHA7YNHGKF4A5PKFNFSTQ, seg_pos={0 0}, upload=458290]: Long tail cancellation due to optimal threshold
Dec 27 09:53:30.104333229 (522h24m29.975416274s) [seg_pos={0 0}, upload=458290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:53:30.104394857 (522h24m29.975477902s) [upload=458290]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:53:30.104399395 (522h24m29.97548243s) [upload=458290]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:53:30.390174408 (522h24m30.261257443s) [upload=458290]: Done uploading segments...
Dec 27 09:53:30.390193194 (522h24m30.261276239s) [upload=458290]: Upload finished. err: <nil>
Duration: 30.002039615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..........X........X.............X........................X.]
duration: 
56.054586954s
trace: 
https://jaeger.dev.storj.io/trace/54fb61fe8ffb1d59
FAIL (2025-12-27 09:53:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, Piece ID: HPTS56RI46YUTRMKNDXC2SCG2FDG4EFUWA3HLL6ZEI7U7E5IX5EQ) context deadline exceeded; piecestore: (Node ID: 1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, Piece ID: G7F7BS345GKEJGYMHANO4JMAXGMUFLGEIHCPTUNHUI475XPJ54BQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: 7NRYXT6W7V24B42LG2FXO2Y7LWAUEC4J2THMOLMSE7CY3JFHP5XA) context deadline exceeded
======LOGS======
 09:51:04.307792764 (522h22m4.178875819s) [noise=true , address=45.91.116.6:4977     , piece_id=HPTS56RI46YUTRMKNDXC2SCG2FDG4EFUWA3HLL6ZEI7U7E5IX5EQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.308268801 (522h22m4.179351836s) [noise=true , address=45.91.116.10:4107    , piece_id=E6QG75NK3APNZVM2LOZTK2MWCYJWQIXAO2VL4J5OP7V6VQB4HDFQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.309424862 (522h22m4.180507897s) [noise=true , address=77.90.31.18:4813     , piece_id=IDWIUJSRCLLV7VT77JX7KFQTHWWJN2XZXTWGYJLMIJREEZ2VD3KQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.309562584 (522h22m4.180645629s) [noise=true , address=77.90.31.11:4562     , piece_id=4Q4PZC7TJA4Q7M6IDXDRIZ3YMPETCS226T2X5SM7I5766WWLBCGQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.310121458 (522h22m4.181204503s) [noise=true , address=145.79.227.30:4770   , piece_id=SFCGMJSM2PWWI6V6CF4QCYRIMXHVF3NEDPPGDSEULAAEVOWVYV3A, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.311117826 (522h22m4.182200861s) [noise=true , address=145.79.227.14:4862   , piece_id=3NYUH5ZJW24B7XWAAG7XUBVCKOK5MZ2RLDORB35U7OZTMB54GPUA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.312194396 (522h22m4.183277441s) [noise=true , address=64.49.13.170:4717    , piece_id=4I5XRBJ7BUHC7KZ75KDTNJ2MKZQKJ4DYVAFRVLED733455IE5QJQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.313185513 (522h22m4.184268558s) [noise=true , address=77.90.31.17:4796     , piece_id=G7F7BS345GKEJGYMHANO4JMAXGMUFLGEIHCPTUNHUI475XPJ54BQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.314531476 (522h22m4.185614521s) [noise=true , address=64.49.13.26:4947     , piece_id=7NRYXT6W7V24B42LG2FXO2Y7LWAUEC4J2THMOLMSE7CY3JFHP5XA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.316003308 (522h22m4.187086353s) [noise=true , address=77.90.31.19:4875     , piece_id=C7ERTU6PJ3JSLDFWXXUY2FACHZ5FHZV444KOVVYXGFEKD5QH26QQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.316553436 (522h22m4.187636471s) [noise=true , address=145.79.227.34:4938   , piece_id=VORSPX32STSZFKYOBC46M7IF6ZCNZTBHGAFVOK52MR4LEYFEVCUQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.319645424 (522h22m4.190728469s) [noise=true , address=145.79.227.35:4513   , piece_id=AX7LU2HBMGHWSMZX3K4KJNG77IEIE7WRXYVVJGZSGAX3EA3MXUQA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.323045277 (522h22m4.194128322s) [noise=true , address=45.91.116.9:4293     , piece_id=JWWRVYXNIEJPEWHSCCN3E4PJLMIA5DMSWICPAEDR6A246PO5ETCA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.323866411 (522h22m4.194949456s) [noise=true , address=64.49.13.169:4706    , piece_id=X4HTGOZYNSLQR6AXFBOAGZJTYJ53C663ZNXSCORWQY7LSMQSUWKA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.325769535 (522h22m4.196852571s) [noise=true , address=45.91.116.8:4073     , piece_id=EDLWJZKKQVPIDK2H77AIOCL6KMQ4OG7N6NXFHIMWTBJQ3Y22ZWPA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.330596025 (522h22m4.20167906s) [noise=true , address=77.90.31.12:4622     , piece_id=PZI76RWBSFFQVOT75IF6SCP3OUTAF6S4DAAD4SHL4HWYTHZQHVPQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:05.646489569 (522h22m5.517572614s) [upload=458255]: Next returned nil segment
Dec 27 09:51:05.646510138 (522h22m5.517593173s) [upload=458255]: Waiting for error group managing segments...
Dec 27 09:51:05.980290448 (522h22m5.851373493s) [noise=true , address=64.49.13.170:4717    , piece_id=4I5XRBJ7BUHC7KZ75KDTNJ2MKZQKJ4DYVAFRVLED733455IE5QJQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.205130765 (522h22m6.07621381s) [noise=true , address=64.49.13.26:4947     , piece_id=7NRYXT6W7V24B42LG2FXO2Y7LWAUEC4J2THMOLMSE7CY3JFHP5XA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.208845889 (522h22m6.079928924s) [noise=true , address=145.79.227.35:4513   , piece_id=AX7LU2HBMGHWSMZX3K4KJNG77IEIE7WRXYVVJGZSGAX3EA3MXUQA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.210837221 (522h22m6.081920266s) [noise=true , address=145.79.227.14:4862   , piece_id=3NYUH5ZJW24B7XWAAG7XUBVCKOK5MZ2RLDORB35U7OZTMB54GPUA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.255553609 (522h22m6.126636654s) [noise=true , address=145.79.227.34:4938   , piece_id=VORSPX32STSZFKYOBC46M7IF6ZCNZTBHGAFVOK52MR4LEYFEVCUQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.384765914 (522h22m6.255848949s) [noise=true , address=77.90.31.19:4875     , piece_id=C7ERTU6PJ3JSLDFWXXUY2FACHZ5FHZV444KOVVYXGFEKD5QH26QQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.423422032 (522h22m6.294505077s) [noise=true , address=45.91.116.6:4977     , piece_id=HPTS56RI46YUTRMKNDXC2SCG2FDG4EFUWA3HLL6ZEI7U7E5IX5EQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.435434556 (522h22m6.306517591s) [noise=true , address=45.91.116.8:4073     , piece_id=EDLWJZKKQVPIDK2H77AIOCL6KMQ4OG7N6NXFHIMWTBJQ3Y22ZWPA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.500209529 (522h22m6.371292564s) [noise=true , address=77.90.31.17:4796     , piece_id=G7F7BS345GKEJGYMHANO4JMAXGMUFLGEIHCPTUNHUI475XPJ54BQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.508276440 (522h22m6.379359475s) [noise=true , address=45.91.116.9:4293     , piece_id=JWWRVYXNIEJPEWHSCCN3E4PJLMIA5DMSWICPAEDR6A246PO5ETCA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.513741266 (522h22m6.384824301s) [noise=true , address=45.91.116.10:4107    , piece_id=E6QG75NK3APNZVM2LOZTK2MWCYJWQIXAO2VL4J5OP7V6VQB4HDFQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.564671116 (522h22m6.435754161s) [noise=true , address=77.90.31.18:4813     , piece_id=IDWIUJSRCLLV7VT77JX7KFQTHWWJN2XZXTWGYJLMIJREEZ2VD3KQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.613260429 (522h22m6.484343474s) [noise=true , address=145.79.227.40:4079   , piece_id=OEOWVY65QQTYQQZLCIEID6LGBOKW4L5EDWOQ7E3UGEKDTWSPIZAQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.802731357 (522h22m6.673814402s) [noise=true , address=77.90.31.11:4562     , piece_id=4Q4PZC7TJA4Q7M6IDXDRIZ3YMPETCS226T2X5SM7I5766WWLBCGQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.802757376 (522h22m6.673840421s) [seg_pos={0 0}, upload=458255]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:51:06.802965072 (522h22m6.674048107s) [noise=true , address=77.90.31.12:4622     , piece_id=PZI76RWBSFFQVOT75IF6SCP3OUTAF6S4DAAD4SHL4HWYTHZQHVPQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MQryCwJkVN1JhFm4QqhypX3WoMDR9rdjELj4oYoVbpa6U823Z): protocol: expected piece hash; context canceled; EOF
Dec 27 09:51:06.802974570 (522h22m6.674057615s) [noise=true , address=77.90.31.12:4622     , piece_id=PZI76RWBSFFQVOT75IF6SCP3OUTAF6S4DAAD4SHL4HWYTHZQHVPQ, seg_pos={0 0}, upload=458255]: Long tail cancellation due to optimal threshold
Dec 27 09:51:06.803135446 (522h22m6.674218491s) [noise=true , address=145.79.227.44:4205   , piece_id=YUEO5KJHEUG45VMFSAFIF6BJPI3K6KMNKYF7Q7V44AL2S7MMDDHQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ): protocol: expected piece hash; context canceled; EOF
Dec 27 09:51:06.803143602 (522h22m6.674226637s) [noise=true , address=145.79.227.44:4205   , piece_id=YUEO5KJHEUG45VMFSAFIF6BJPI3K6KMNKYF7Q7V44AL2S7MMDDHQ, seg_pos={0 0}, upload=458255]: Long tail cancellation due to optimal threshold
Dec 27 09:51:06.803157969 (522h22m6.674241014s) [noise=true , address=145.79.227.30:4770   , piece_id=SFCGMJSM2PWWI6V6CF4QCYRIMXHVF3NEDPPGDSEULAAEVOWVYV3A, seg_pos={0 0}, upload=458255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF
Dec 27 09:51:06.803167157 (522h22m6.674250192s) [noise=true , address=145.79.227.30:4770   , piece_id=SFCGMJSM2PWWI6V6CF4QCYRIMXHVF3NEDPPGDSEULAAEVOWVYV3A, seg_pos={0 0}, upload=458255]: Long tail cancellation due to optimal threshold
Dec 27 09:51:06.803714098 (522h22m6.674797133s) [noise=true , address=64.49.13.169:4706    , piece_id=X4HTGOZYNSLQR6AXFBOAGZJTYJ53C663ZNXSCORWQY7LSMQSUWKA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:53990->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:53990->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53990->64.49.13.169:4706: use of closed network connection
Dec 27 09:51:06.803727063 (522h22m6.674810098s) [noise=true , address=64.49.13.169:4706    , piece_id=X4HTGOZYNSLQR6AXFBOAGZJTYJ53C663ZNXSCORWQY7LSMQSUWKA, seg_pos={0 0}, upload=458255]: Long tail cancellation due to optimal threshold
Dec 27 09:51:06.803792778 (522h22m6.674875823s) [seg_pos={0 0}, upload=458255]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:51:06.803876578 (522h22m6.674959613s) [upload=458255]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:51:06.803897026 (522h22m6.674980071s) [upload=458255]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:51:07.108514183 (522h22m6.979597228s) [upload=458255]: Done uploading segments...
Dec 27 09:51:07.108552437 (522h22m6.979635472s) [upload=458255]: Upload finished. err: <nil>
Duration: 56.054615979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.....]
duration: 
5.679833415s
trace: 
https://jaeger.dev.storj.io/trace/15416150393d7cce
FAIL (2025-12-27 09:52:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 09:49:14.536808671 (522h20m14.407891716s) [upload=458226]: Starting upload
Dec 27 09:49:14.536872493 (522h20m14.407955538s) [upload=458226]: Done starting upload
Dec 27 09:49:14.536987291 (522h20m14.408070326s) [upload=458226]: Uploading segments...
Dec 27 09:49:14.537089005 (522h20m14.40817204s) [upload=458226]: Got next segment. Inline: false
Dec 27 09:49:14.537120374 (522h20m14.40820341s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:49:14.537124913 (522h20m14.408207948s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:49:14.778069075 (522h20m14.64915211s) [seg_pos={0 0}, upload=458226]: Begin upload segment...
Dec 27 09:49:14.778112198 (522h20m14.649195233s) [seg_pos={0 0}, upload=458226]: Done begin upload segment.
Dec 27 09:49:14.778139990 (522h20m14.649223025s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778147224 (522h20m14.649230269s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778688184 (522h20m14.649771229s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779025066 (522h20m14.650108101s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779304448 (522h20m14.650387493s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779637833 (522h20m14.650720878s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779975365 (522h20m14.6510584s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780259747 (522h20m14.651342782s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780598862 (522h20m14.651681897s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781101991 (522h20m14.652185036s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781593657 (522h20m14.652676702s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783054579 (522h20m14.654137624s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783987996 (522h20m14.655071031s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.786792216 (522h20m14.657875251s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.787952846 (522h20m14.659035881s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.789398949 (522h20m14.660481984s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:16.489507881 (522h20m16.360590916s) [upload=458226]: Next returned nil segment
Dec 27 09:49:16.489521115 (522h20m16.36060416s) [upload=458226]: Waiting for error group managing segments...
Dec 27 09:49:16.777390442 (522h20m16.648473477s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.790016855 (522h20m16.66109989s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.805443389 (522h20m16.676526425s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.810805931 (522h20m16.681888966s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.813964956 (522h20m16.685048001s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.824629043 (522h20m16.695712088s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.971709331 (522h20m16.842792366s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.975405509 (522h20m16.846488544s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.017105353 (522h20m16.888188388s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.064717586 (522h20m16.935800621s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.088446329 (522h20m16.959529374s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.149932731 (522h20m17.021015776s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659586856 (522h20m17.530669891s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659604490 (522h20m17.530687525s) [seg_pos={0 0}, upload=458226]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:49:17.659877951 (522h20m17.530960996s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659888972 (522h20m17.530972027s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.659962041 (522h20m17.531045076s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659990485 (522h20m17.53107353s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660082210 (522h20m17.531165255s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.660091367 (522h20m17.531174413s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660111876 (522h20m17.531194921s) [seg_pos={0 0}, upload=458226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:49:17.660183833 (522h20m17.531266868s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:49:17.660189535 (522h20m17.53127257s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:49:17.940812001 (522h20m17.811895036s) [upload=458226]: Done uploading segments...
Dec 27 09:49:17.940832049 (522h20m17.811915084s) [upload=458226]: Upload finished. err: <nil>
Duration: 5.679861416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX....]
duration: 
5.823893565s
trace: 
https://jaeger.dev.storj.io/trace/15416150393d7cce
FAIL (2025-12-27 09:52:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 09:49:14.536808671 (522h20m14.407891716s) [upload=458226]: Starting upload
Dec 27 09:49:14.536872493 (522h20m14.407955538s) [upload=458226]: Done starting upload
Dec 27 09:49:14.536987291 (522h20m14.408070326s) [upload=458226]: Uploading segments...
Dec 27 09:49:14.537089005 (522h20m14.40817204s) [upload=458226]: Got next segment. Inline: false
Dec 27 09:49:14.537120374 (522h20m14.40820341s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:49:14.537124913 (522h20m14.408207948s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:49:14.778069075 (522h20m14.64915211s) [seg_pos={0 0}, upload=458226]: Begin upload segment...
Dec 27 09:49:14.778112198 (522h20m14.649195233s) [seg_pos={0 0}, upload=458226]: Done begin upload segment.
Dec 27 09:49:14.778139990 (522h20m14.649223025s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778147224 (522h20m14.649230269s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778688184 (522h20m14.649771229s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779025066 (522h20m14.650108101s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779304448 (522h20m14.650387493s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779637833 (522h20m14.650720878s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779975365 (522h20m14.6510584s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780259747 (522h20m14.651342782s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780598862 (522h20m14.651681897s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781101991 (522h20m14.652185036s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781593657 (522h20m14.652676702s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783054579 (522h20m14.654137624s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783987996 (522h20m14.655071031s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.786792216 (522h20m14.657875251s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.787952846 (522h20m14.659035881s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.789398949 (522h20m14.660481984s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:16.489507881 (522h20m16.360590916s) [upload=458226]: Next returned nil segment
Dec 27 09:49:16.489521115 (522h20m16.36060416s) [upload=458226]: Waiting for error group managing segments...
Dec 27 09:49:16.777390442 (522h20m16.648473477s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.790016855 (522h20m16.66109989s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.805443389 (522h20m16.676526425s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.810805931 (522h20m16.681888966s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.813964956 (522h20m16.685048001s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.824629043 (522h20m16.695712088s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.971709331 (522h20m16.842792366s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.975405509 (522h20m16.846488544s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.017105353 (522h20m16.888188388s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.064717586 (522h20m16.935800621s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.088446329 (522h20m16.959529374s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.149932731 (522h20m17.021015776s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659586856 (522h20m17.530669891s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659604490 (522h20m17.530687525s) [seg_pos={0 0}, upload=458226]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:49:17.659877951 (522h20m17.530960996s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659888972 (522h20m17.530972027s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.659962041 (522h20m17.531045076s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659990485 (522h20m17.53107353s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660082210 (522h20m17.531165255s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.660091367 (522h20m17.531174413s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660111876 (522h20m17.531194921s) [seg_pos={0 0}, upload=458226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:49:17.660183833 (522h20m17.531266868s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:49:17.660189535 (522h20m17.53127257s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:49:17.940812001 (522h20m17.811895036s) [upload=458226]: Done uploading segments...
Dec 27 09:49:17.940832049 (522h20m17.811915084s) [upload=458226]: Upload finished. err: <nil>
Duration: 5.823920407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [...........X........X.............X........................X]
duration: 
1m0.003030266s
trace: 
https://jaeger.dev.storj.io/trace/54fb61fe8ffb1d59
FAIL (2025-12-27 09:52:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, Piece ID: HPTS56RI46YUTRMKNDXC2SCG2FDG4EFUWA3HLL6ZEI7U7E5IX5EQ) context deadline exceeded; piecestore: (Node ID: 1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, Piece ID: G7F7BS345GKEJGYMHANO4JMAXGMUFLGEIHCPTUNHUI475XPJ54BQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: 7NRYXT6W7V24B42LG2FXO2Y7LWAUEC4J2THMOLMSE7CY3JFHP5XA) context deadline exceeded
======LOGS======
 09:51:04.307792764 (522h22m4.178875819s) [noise=true , address=45.91.116.6:4977     , piece_id=HPTS56RI46YUTRMKNDXC2SCG2FDG4EFUWA3HLL6ZEI7U7E5IX5EQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.308268801 (522h22m4.179351836s) [noise=true , address=45.91.116.10:4107    , piece_id=E6QG75NK3APNZVM2LOZTK2MWCYJWQIXAO2VL4J5OP7V6VQB4HDFQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.309424862 (522h22m4.180507897s) [noise=true , address=77.90.31.18:4813     , piece_id=IDWIUJSRCLLV7VT77JX7KFQTHWWJN2XZXTWGYJLMIJREEZ2VD3KQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.309562584 (522h22m4.180645629s) [noise=true , address=77.90.31.11:4562     , piece_id=4Q4PZC7TJA4Q7M6IDXDRIZ3YMPETCS226T2X5SM7I5766WWLBCGQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.310121458 (522h22m4.181204503s) [noise=true , address=145.79.227.30:4770   , piece_id=SFCGMJSM2PWWI6V6CF4QCYRIMXHVF3NEDPPGDSEULAAEVOWVYV3A, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.311117826 (522h22m4.182200861s) [noise=true , address=145.79.227.14:4862   , piece_id=3NYUH5ZJW24B7XWAAG7XUBVCKOK5MZ2RLDORB35U7OZTMB54GPUA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.312194396 (522h22m4.183277441s) [noise=true , address=64.49.13.170:4717    , piece_id=4I5XRBJ7BUHC7KZ75KDTNJ2MKZQKJ4DYVAFRVLED733455IE5QJQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.313185513 (522h22m4.184268558s) [noise=true , address=77.90.31.17:4796     , piece_id=G7F7BS345GKEJGYMHANO4JMAXGMUFLGEIHCPTUNHUI475XPJ54BQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.314531476 (522h22m4.185614521s) [noise=true , address=64.49.13.26:4947     , piece_id=7NRYXT6W7V24B42LG2FXO2Y7LWAUEC4J2THMOLMSE7CY3JFHP5XA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.316003308 (522h22m4.187086353s) [noise=true , address=77.90.31.19:4875     , piece_id=C7ERTU6PJ3JSLDFWXXUY2FACHZ5FHZV444KOVVYXGFEKD5QH26QQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.316553436 (522h22m4.187636471s) [noise=true , address=145.79.227.34:4938   , piece_id=VORSPX32STSZFKYOBC46M7IF6ZCNZTBHGAFVOK52MR4LEYFEVCUQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.319645424 (522h22m4.190728469s) [noise=true , address=145.79.227.35:4513   , piece_id=AX7LU2HBMGHWSMZX3K4KJNG77IEIE7WRXYVVJGZSGAX3EA3MXUQA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.323045277 (522h22m4.194128322s) [noise=true , address=45.91.116.9:4293     , piece_id=JWWRVYXNIEJPEWHSCCN3E4PJLMIA5DMSWICPAEDR6A246PO5ETCA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.323866411 (522h22m4.194949456s) [noise=true , address=64.49.13.169:4706    , piece_id=X4HTGOZYNSLQR6AXFBOAGZJTYJ53C663ZNXSCORWQY7LSMQSUWKA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.325769535 (522h22m4.196852571s) [noise=true , address=45.91.116.8:4073     , piece_id=EDLWJZKKQVPIDK2H77AIOCL6KMQ4OG7N6NXFHIMWTBJQ3Y22ZWPA, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:04.330596025 (522h22m4.20167906s) [noise=true , address=77.90.31.12:4622     , piece_id=PZI76RWBSFFQVOT75IF6SCP3OUTAF6S4DAAD4SHL4HWYTHZQHVPQ, seg_pos={0 0}, upload=458255]: Uploading piece...
Dec 27 09:51:05.646489569 (522h22m5.517572614s) [upload=458255]: Next returned nil segment
Dec 27 09:51:05.646510138 (522h22m5.517593173s) [upload=458255]: Waiting for error group managing segments...
Dec 27 09:51:05.980290448 (522h22m5.851373493s) [noise=true , address=64.49.13.170:4717    , piece_id=4I5XRBJ7BUHC7KZ75KDTNJ2MKZQKJ4DYVAFRVLED733455IE5QJQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.205130765 (522h22m6.07621381s) [noise=true , address=64.49.13.26:4947     , piece_id=7NRYXT6W7V24B42LG2FXO2Y7LWAUEC4J2THMOLMSE7CY3JFHP5XA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.208845889 (522h22m6.079928924s) [noise=true , address=145.79.227.35:4513   , piece_id=AX7LU2HBMGHWSMZX3K4KJNG77IEIE7WRXYVVJGZSGAX3EA3MXUQA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.210837221 (522h22m6.081920266s) [noise=true , address=145.79.227.14:4862   , piece_id=3NYUH5ZJW24B7XWAAG7XUBVCKOK5MZ2RLDORB35U7OZTMB54GPUA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.255553609 (522h22m6.126636654s) [noise=true , address=145.79.227.34:4938   , piece_id=VORSPX32STSZFKYOBC46M7IF6ZCNZTBHGAFVOK52MR4LEYFEVCUQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.384765914 (522h22m6.255848949s) [noise=true , address=77.90.31.19:4875     , piece_id=C7ERTU6PJ3JSLDFWXXUY2FACHZ5FHZV444KOVVYXGFEKD5QH26QQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.423422032 (522h22m6.294505077s) [noise=true , address=45.91.116.6:4977     , piece_id=HPTS56RI46YUTRMKNDXC2SCG2FDG4EFUWA3HLL6ZEI7U7E5IX5EQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.435434556 (522h22m6.306517591s) [noise=true , address=45.91.116.8:4073     , piece_id=EDLWJZKKQVPIDK2H77AIOCL6KMQ4OG7N6NXFHIMWTBJQ3Y22ZWPA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.500209529 (522h22m6.371292564s) [noise=true , address=77.90.31.17:4796     , piece_id=G7F7BS345GKEJGYMHANO4JMAXGMUFLGEIHCPTUNHUI475XPJ54BQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.508276440 (522h22m6.379359475s) [noise=true , address=45.91.116.9:4293     , piece_id=JWWRVYXNIEJPEWHSCCN3E4PJLMIA5DMSWICPAEDR6A246PO5ETCA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.513741266 (522h22m6.384824301s) [noise=true , address=45.91.116.10:4107    , piece_id=E6QG75NK3APNZVM2LOZTK2MWCYJWQIXAO2VL4J5OP7V6VQB4HDFQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.564671116 (522h22m6.435754161s) [noise=true , address=77.90.31.18:4813     , piece_id=IDWIUJSRCLLV7VT77JX7KFQTHWWJN2XZXTWGYJLMIJREEZ2VD3KQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.613260429 (522h22m6.484343474s) [noise=true , address=145.79.227.40:4079   , piece_id=OEOWVY65QQTYQQZLCIEID6LGBOKW4L5EDWOQ7E3UGEKDTWSPIZAQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.802731357 (522h22m6.673814402s) [noise=true , address=77.90.31.11:4562     , piece_id=4Q4PZC7TJA4Q7M6IDXDRIZ3YMPETCS226T2X5SM7I5766WWLBCGQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: <nil>
Dec 27 09:51:06.802757376 (522h22m6.673840421s) [seg_pos={0 0}, upload=458255]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:51:06.802965072 (522h22m6.674048107s) [noise=true , address=77.90.31.12:4622     , piece_id=PZI76RWBSFFQVOT75IF6SCP3OUTAF6S4DAAD4SHL4HWYTHZQHVPQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MQryCwJkVN1JhFm4QqhypX3WoMDR9rdjELj4oYoVbpa6U823Z): protocol: expected piece hash; context canceled; EOF
Dec 27 09:51:06.802974570 (522h22m6.674057615s) [noise=true , address=77.90.31.12:4622     , piece_id=PZI76RWBSFFQVOT75IF6SCP3OUTAF6S4DAAD4SHL4HWYTHZQHVPQ, seg_pos={0 0}, upload=458255]: Long tail cancellation due to optimal threshold
Dec 27 09:51:06.803135446 (522h22m6.674218491s) [noise=true , address=145.79.227.44:4205   , piece_id=YUEO5KJHEUG45VMFSAFIF6BJPI3K6KMNKYF7Q7V44AL2S7MMDDHQ, seg_pos={0 0}, upload=458255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ): protocol: expected piece hash; context canceled; EOF
Dec 27 09:51:06.803143602 (522h22m6.674226637s) [noise=true , address=145.79.227.44:4205   , piece_id=YUEO5KJHEUG45VMFSAFIF6BJPI3K6KMNKYF7Q7V44AL2S7MMDDHQ, seg_pos={0 0}, upload=458255]: Long tail cancellation due to optimal threshold
Dec 27 09:51:06.803157969 (522h22m6.674241014s) [noise=true , address=145.79.227.30:4770   , piece_id=SFCGMJSM2PWWI6V6CF4QCYRIMXHVF3NEDPPGDSEULAAEVOWVYV3A, seg_pos={0 0}, upload=458255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF
Dec 27 09:51:06.803167157 (522h22m6.674250192s) [noise=true , address=145.79.227.30:4770   , piece_id=SFCGMJSM2PWWI6V6CF4QCYRIMXHVF3NEDPPGDSEULAAEVOWVYV3A, seg_pos={0 0}, upload=458255]: Long tail cancellation due to optimal threshold
Dec 27 09:51:06.803714098 (522h22m6.674797133s) [noise=true , address=64.49.13.169:4706    , piece_id=X4HTGOZYNSLQR6AXFBOAGZJTYJ53C663ZNXSCORWQY7LSMQSUWKA, seg_pos={0 0}, upload=458255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:53990->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:53990->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53990->64.49.13.169:4706: use of closed network connection
Dec 27 09:51:06.803727063 (522h22m6.674810098s) [noise=true , address=64.49.13.169:4706    , piece_id=X4HTGOZYNSLQR6AXFBOAGZJTYJ53C663ZNXSCORWQY7LSMQSUWKA, seg_pos={0 0}, upload=458255]: Long tail cancellation due to optimal threshold
Dec 27 09:51:06.803792778 (522h22m6.674875823s) [seg_pos={0 0}, upload=458255]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:51:06.803876578 (522h22m6.674959613s) [upload=458255]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:51:06.803897026 (522h22m6.674980071s) [upload=458255]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:51:07.108514183 (522h22m6.979597228s) [upload=458255]: Done uploading segments...
Dec 27 09:51:07.108552437 (522h22m6.979635472s) [upload=458255]: Upload finished. err: <nil>
Duration: 1m0.003058691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX...]
duration: 
5.87094838s
trace: 
https://jaeger.dev.storj.io/trace/15416150393d7cce
FAIL (2025-12-27 09:51:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 09:49:14.536808671 (522h20m14.407891716s) [upload=458226]: Starting upload
Dec 27 09:49:14.536872493 (522h20m14.407955538s) [upload=458226]: Done starting upload
Dec 27 09:49:14.536987291 (522h20m14.408070326s) [upload=458226]: Uploading segments...
Dec 27 09:49:14.537089005 (522h20m14.40817204s) [upload=458226]: Got next segment. Inline: false
Dec 27 09:49:14.537120374 (522h20m14.40820341s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:49:14.537124913 (522h20m14.408207948s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:49:14.778069075 (522h20m14.64915211s) [seg_pos={0 0}, upload=458226]: Begin upload segment...
Dec 27 09:49:14.778112198 (522h20m14.649195233s) [seg_pos={0 0}, upload=458226]: Done begin upload segment.
Dec 27 09:49:14.778139990 (522h20m14.649223025s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778147224 (522h20m14.649230269s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778688184 (522h20m14.649771229s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779025066 (522h20m14.650108101s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779304448 (522h20m14.650387493s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779637833 (522h20m14.650720878s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779975365 (522h20m14.6510584s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780259747 (522h20m14.651342782s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780598862 (522h20m14.651681897s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781101991 (522h20m14.652185036s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781593657 (522h20m14.652676702s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783054579 (522h20m14.654137624s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783987996 (522h20m14.655071031s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.786792216 (522h20m14.657875251s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.787952846 (522h20m14.659035881s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.789398949 (522h20m14.660481984s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:16.489507881 (522h20m16.360590916s) [upload=458226]: Next returned nil segment
Dec 27 09:49:16.489521115 (522h20m16.36060416s) [upload=458226]: Waiting for error group managing segments...
Dec 27 09:49:16.777390442 (522h20m16.648473477s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.790016855 (522h20m16.66109989s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.805443389 (522h20m16.676526425s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.810805931 (522h20m16.681888966s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.813964956 (522h20m16.685048001s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.824629043 (522h20m16.695712088s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.971709331 (522h20m16.842792366s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.975405509 (522h20m16.846488544s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.017105353 (522h20m16.888188388s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.064717586 (522h20m16.935800621s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.088446329 (522h20m16.959529374s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.149932731 (522h20m17.021015776s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659586856 (522h20m17.530669891s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659604490 (522h20m17.530687525s) [seg_pos={0 0}, upload=458226]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:49:17.659877951 (522h20m17.530960996s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659888972 (522h20m17.530972027s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.659962041 (522h20m17.531045076s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659990485 (522h20m17.53107353s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660082210 (522h20m17.531165255s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.660091367 (522h20m17.531174413s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660111876 (522h20m17.531194921s) [seg_pos={0 0}, upload=458226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:49:17.660183833 (522h20m17.531266868s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:49:17.660189535 (522h20m17.53127257s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:49:17.940812001 (522h20m17.811895036s) [upload=458226]: Done uploading segments...
Dec 27 09:49:17.940832049 (522h20m17.811915084s) [upload=458226]: Upload finished. err: <nil>
Duration: 5.870978468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [............X........X.............X........................]
duration: 
41.711651266s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:50:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 41.71166859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [.X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX..]
duration: 
6.31164384s
trace: 
https://jaeger.dev.storj.io/trace/15416150393d7cce
FAIL (2025-12-27 09:50:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 09:49:14.536808671 (522h20m14.407891716s) [upload=458226]: Starting upload
Dec 27 09:49:14.536872493 (522h20m14.407955538s) [upload=458226]: Done starting upload
Dec 27 09:49:14.536987291 (522h20m14.408070326s) [upload=458226]: Uploading segments...
Dec 27 09:49:14.537089005 (522h20m14.40817204s) [upload=458226]: Got next segment. Inline: false
Dec 27 09:49:14.537120374 (522h20m14.40820341s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:49:14.537124913 (522h20m14.408207948s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:49:14.778069075 (522h20m14.64915211s) [seg_pos={0 0}, upload=458226]: Begin upload segment...
Dec 27 09:49:14.778112198 (522h20m14.649195233s) [seg_pos={0 0}, upload=458226]: Done begin upload segment.
Dec 27 09:49:14.778139990 (522h20m14.649223025s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778147224 (522h20m14.649230269s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778688184 (522h20m14.649771229s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779025066 (522h20m14.650108101s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779304448 (522h20m14.650387493s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779637833 (522h20m14.650720878s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779975365 (522h20m14.6510584s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780259747 (522h20m14.651342782s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780598862 (522h20m14.651681897s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781101991 (522h20m14.652185036s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781593657 (522h20m14.652676702s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783054579 (522h20m14.654137624s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783987996 (522h20m14.655071031s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.786792216 (522h20m14.657875251s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.787952846 (522h20m14.659035881s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.789398949 (522h20m14.660481984s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:16.489507881 (522h20m16.360590916s) [upload=458226]: Next returned nil segment
Dec 27 09:49:16.489521115 (522h20m16.36060416s) [upload=458226]: Waiting for error group managing segments...
Dec 27 09:49:16.777390442 (522h20m16.648473477s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.790016855 (522h20m16.66109989s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.805443389 (522h20m16.676526425s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.810805931 (522h20m16.681888966s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.813964956 (522h20m16.685048001s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.824629043 (522h20m16.695712088s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.971709331 (522h20m16.842792366s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.975405509 (522h20m16.846488544s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.017105353 (522h20m16.888188388s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.064717586 (522h20m16.935800621s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.088446329 (522h20m16.959529374s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.149932731 (522h20m17.021015776s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659586856 (522h20m17.530669891s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659604490 (522h20m17.530687525s) [seg_pos={0 0}, upload=458226]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:49:17.659877951 (522h20m17.530960996s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659888972 (522h20m17.530972027s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.659962041 (522h20m17.531045076s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659990485 (522h20m17.53107353s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660082210 (522h20m17.531165255s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.660091367 (522h20m17.531174413s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660111876 (522h20m17.531194921s) [seg_pos={0 0}, upload=458226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:49:17.660183833 (522h20m17.531266868s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:49:17.660189535 (522h20m17.53127257s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:49:17.940812001 (522h20m17.811895036s) [upload=458226]: Done uploading segments...
Dec 27 09:49:17.940832049 (522h20m17.811915084s) [upload=458226]: Upload finished. err: <nil>
Duration: 6.311672784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.............X........X.............X.......................]
duration: 
47.192203173s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:50:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 47.192221767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [..X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX.]
duration: 
5.972896209s
trace: 
https://jaeger.dev.storj.io/trace/15416150393d7cce
FAIL (2025-12-27 09:50:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 09:49:14.536808671 (522h20m14.407891716s) [upload=458226]: Starting upload
Dec 27 09:49:14.536872493 (522h20m14.407955538s) [upload=458226]: Done starting upload
Dec 27 09:49:14.536987291 (522h20m14.408070326s) [upload=458226]: Uploading segments...
Dec 27 09:49:14.537089005 (522h20m14.40817204s) [upload=458226]: Got next segment. Inline: false
Dec 27 09:49:14.537120374 (522h20m14.40820341s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:49:14.537124913 (522h20m14.408207948s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:49:14.778069075 (522h20m14.64915211s) [seg_pos={0 0}, upload=458226]: Begin upload segment...
Dec 27 09:49:14.778112198 (522h20m14.649195233s) [seg_pos={0 0}, upload=458226]: Done begin upload segment.
Dec 27 09:49:14.778139990 (522h20m14.649223025s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778147224 (522h20m14.649230269s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778688184 (522h20m14.649771229s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779025066 (522h20m14.650108101s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779304448 (522h20m14.650387493s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779637833 (522h20m14.650720878s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779975365 (522h20m14.6510584s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780259747 (522h20m14.651342782s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780598862 (522h20m14.651681897s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781101991 (522h20m14.652185036s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781593657 (522h20m14.652676702s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783054579 (522h20m14.654137624s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783987996 (522h20m14.655071031s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.786792216 (522h20m14.657875251s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.787952846 (522h20m14.659035881s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.789398949 (522h20m14.660481984s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:16.489507881 (522h20m16.360590916s) [upload=458226]: Next returned nil segment
Dec 27 09:49:16.489521115 (522h20m16.36060416s) [upload=458226]: Waiting for error group managing segments...
Dec 27 09:49:16.777390442 (522h20m16.648473477s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.790016855 (522h20m16.66109989s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.805443389 (522h20m16.676526425s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.810805931 (522h20m16.681888966s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.813964956 (522h20m16.685048001s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.824629043 (522h20m16.695712088s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.971709331 (522h20m16.842792366s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.975405509 (522h20m16.846488544s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.017105353 (522h20m16.888188388s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.064717586 (522h20m16.935800621s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.088446329 (522h20m16.959529374s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.149932731 (522h20m17.021015776s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659586856 (522h20m17.530669891s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659604490 (522h20m17.530687525s) [seg_pos={0 0}, upload=458226]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:49:17.659877951 (522h20m17.530960996s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659888972 (522h20m17.530972027s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.659962041 (522h20m17.531045076s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659990485 (522h20m17.53107353s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660082210 (522h20m17.531165255s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.660091367 (522h20m17.531174413s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660111876 (522h20m17.531194921s) [seg_pos={0 0}, upload=458226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:49:17.660183833 (522h20m17.531266868s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:49:17.660189535 (522h20m17.53127257s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:49:17.940812001 (522h20m17.811895036s) [upload=458226]: Done uploading segments...
Dec 27 09:49:17.940832049 (522h20m17.811915084s) [upload=458226]: Upload finished. err: <nil>
Duration: 5.972926466s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [...X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XXX]
duration: 
30.002675918s
trace: 
https://jaeger.dev.storj.io/trace/15416150393d7cce
FAIL (2025-12-27 09:49:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 09:49:14.536808671 (522h20m14.407891716s) [upload=458226]: Starting upload
Dec 27 09:49:14.536872493 (522h20m14.407955538s) [upload=458226]: Done starting upload
Dec 27 09:49:14.536987291 (522h20m14.408070326s) [upload=458226]: Uploading segments...
Dec 27 09:49:14.537089005 (522h20m14.40817204s) [upload=458226]: Got next segment. Inline: false
Dec 27 09:49:14.537120374 (522h20m14.40820341s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:49:14.537124913 (522h20m14.408207948s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:49:14.778069075 (522h20m14.64915211s) [seg_pos={0 0}, upload=458226]: Begin upload segment...
Dec 27 09:49:14.778112198 (522h20m14.649195233s) [seg_pos={0 0}, upload=458226]: Done begin upload segment.
Dec 27 09:49:14.778139990 (522h20m14.649223025s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778147224 (522h20m14.649230269s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.778688184 (522h20m14.649771229s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779025066 (522h20m14.650108101s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779304448 (522h20m14.650387493s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779637833 (522h20m14.650720878s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.779975365 (522h20m14.6510584s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780259747 (522h20m14.651342782s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.780598862 (522h20m14.651681897s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781101991 (522h20m14.652185036s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.781593657 (522h20m14.652676702s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783054579 (522h20m14.654137624s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.783987996 (522h20m14.655071031s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.786792216 (522h20m14.657875251s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.787952846 (522h20m14.659035881s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:14.789398949 (522h20m14.660481984s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Uploading piece...
Dec 27 09:49:16.489507881 (522h20m16.360590916s) [upload=458226]: Next returned nil segment
Dec 27 09:49:16.489521115 (522h20m16.36060416s) [upload=458226]: Waiting for error group managing segments...
Dec 27 09:49:16.777390442 (522h20m16.648473477s) [noise=true , address=145.79.227.23:4546   , piece_id=MTQRLB7KBUJ2LEW6KZEGG2C2MS2O6PCQKT4353OKKIXUD5TXLIUQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.790016855 (522h20m16.66109989s) [noise=true , address=145.79.227.45:4230   , piece_id=UPQYPNA3XQENFS42N472WD7IQFP2RRJGHT6ZL2O2IZMAE7OKZO4Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.805443389 (522h20m16.676526425s) [noise=true , address=145.79.227.38:4979   , piece_id=SGKM2MWGJXDGICVKPX2RMZAPP3WT56B4F4KDHLBAIJFCWJWPMOTQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.810805931 (522h20m16.681888966s) [noise=true , address=145.79.227.17:4971   , piece_id=DQVIFTWUX4PGWECLNCDVWYL735A7XKWQY4EJMVBN6R3LFR52P4KA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.813964956 (522h20m16.685048001s) [noise=true , address=145.79.227.25:4618   , piece_id=BWA5P7YDRLBMCHMVAYDC5FX52VOO4VJCOMS2DWOBJAEC6DFGJ7MA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.824629043 (522h20m16.695712088s) [noise=true , address=145.79.227.47:4265   , piece_id=PCED7SYV4HFC6F4IBXACTILKTQ6DUJIVEQMIUF7X6OBU37CLHWRQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.971709331 (522h20m16.842792366s) [noise=true , address=77.90.31.15:4700     , piece_id=OCJ743HFONOX62ONJRCX26VI2BSKSHXPKONL7VJM6HZLRARCZRJA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:16.975405509 (522h20m16.846488544s) [noise=true , address=77.90.31.16:4735     , piece_id=GSXOO5F6WAH3DQ6YOJNGWUH7DMXA7IFIOPPVTL3XGETJMMUAT4LQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.017105353 (522h20m16.888188388s) [noise=true , address=45.91.116.11:4119    , piece_id=BO4F5XT5LBOSF2LZZ2VZDOWEC52DFD4BQ5GI3334IQT6SC3ZKC5Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.064717586 (522h20m16.935800621s) [noise=true , address=45.91.116.13:4156    , piece_id=LHOWMPEJBSPIIGZW5CDQRWNEMVVCHUVVP7LKFU3FQO5VMAJC3BLA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.088446329 (522h20m16.959529374s) [noise=true , address=77.90.31.18:4805     , piece_id=DP53RLN6VDUOZTKRX3ZOYGBCZPIZWURJ47JZD5DRQJZGQJXBI62A, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.149932731 (522h20m17.021015776s) [noise=true , address=45.91.116.6:4975     , piece_id=M5O2VI5LOEZIULASTJ5RVIZOEIDOQHYPHH2KRBLD2NHUYYGCAEWQ, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659586856 (522h20m17.530669891s) [noise=true , address=45.91.116.7:4012     , piece_id=LP7FKCLDMA6CFY6YYAIAMC5TLD3P557PZIJ6FWO7BDRAN5AOMQGA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: <nil>
Dec 27 09:49:17.659604490 (522h20m17.530687525s) [seg_pos={0 0}, upload=458226]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:49:17.659877951 (522h20m17.530960996s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659888972 (522h20m17.530972027s) [noise=true , address=45.91.116.10:4084    , piece_id=5ZXD62LSRMX5ZCR35U5MYRDGLPHEBR4V4EJDMVFRX3XCA7FVEFPA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.659962041 (522h20m17.531045076s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.659990485 (522h20m17.53107353s) [noise=true , address=77.90.31.13:4627     , piece_id=DSI457NIWP2TS45VXGKVNIZNG2K3VEF7JU76PBUS5HBPBMAHAG2Q, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660082210 (522h20m17.531165255s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 09:49:17.660091367 (522h20m17.531174413s) [noise=true , address=45.91.116.9:4261     , piece_id=B5HXZHLROKYXHMJQ3YSJYU3CD5CL64S73YU76UYTNZ4Z3CJGDECA, seg_pos={0 0}, upload=458226]: Long tail cancellation due to optimal threshold
Dec 27 09:49:17.660111876 (522h20m17.531194921s) [seg_pos={0 0}, upload=458226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:49:17.660183833 (522h20m17.531266868s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:49:17.660189535 (522h20m17.53127257s) [upload=458226]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:49:17.940812001 (522h20m17.811895036s) [upload=458226]: Done uploading segments...
Dec 27 09:49:17.940832049 (522h20m17.811915084s) [upload=458226]: Upload finished. err: <nil>
Duration: 30.002696607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..............X........X.............X......................]
duration: 
58.120728901s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:49:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 58.120745592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [....X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..XX]
duration: 
30.011333509s
trace: 
https://jaeger.dev.storj.io/trace/7cded754937eca34
FAIL (2025-12-27 09:48:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: AV2GGH3SIEKBKX4Y2ZC73JK4CWOVYQZ4VFN4VOO2DHG44K34RIUA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: AAS4TYSEJ34Y5Y5VBC4GYOR3SXHV5YBIOCIQGTDCDSIESN3ZIYSA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DBUCWDFWWBHVMXGEQDVUIWC7SDKVIEXWX2MRGXV2AU5JEYBAFKEA) context deadline exceeded
======LOGS======
Dec 27 09:48:08.381198028 (522h19m8.252281063s) [upload=458210]: Done starting upload
Dec 27 09:48:08.381246269 (522h19m8.252329304s) [upload=458210]: Uploading segments...
Dec 27 09:48:08.381389853 (522h19m8.252472888s) [upload=458210]: Got next segment. Inline: false
Dec 27 09:48:08.381424799 (522h19m8.252507834s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:48:08.381428767 (522h19m8.252511812s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:48:08.624812965 (522h19m8.495896s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 27 09:48:08.624859664 (522h19m8.495942719s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 27 09:48:08.624887838 (522h19m8.495970883s) [noise=true , address=145.79.227.25:4617   , piece_id=IKEKZW3RXK7FLT554SBL7VVYUIKAN7NMRCD2ZMEK4MG7MMOBBRRA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.625054946 (522h19m8.496137991s) [noise=true , address=145.79.227.52:4447   , piece_id=RG4HQK2NVM7OFJI7ENCCVIT3UVPV6JH3H7ZPJJOGYIRFJMSUFUJA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.626034091 (522h19m8.497117126s) [noise=true , address=145.79.227.38:4978   , piece_id=EHERKEMYPM3KZFIEGIFVUVHDYQ55IOFAZCLBWO7AB34ZFVINGZLA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.626496311 (522h19m8.497579357s) [noise=true , address=145.79.227.23:4546   , piece_id=5I5NAETOGAE2HJL2QUZEOUNFEOQTEZNWUXX3POFPGUM4RB27EMDQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.626934916 (522h19m8.498017961s) [noise=true , address=145.79.227.33:4871   , piece_id=MAYAQZK7SFMPEBARABRWQFH54ERV6XMFTBFN6LHOZNUI3Z42ROMQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.627362530 (522h19m8.498445565s) [noise=true , address=77.90.31.21:4877     , piece_id=W6MBKMNQKPS5B6D26D3QND6NBNC3JPRZWNK636DBU2LD3O2EX4VQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.628631186 (522h19m8.499714221s) [noise=true , address=145.79.227.14:4840   , piece_id=5C7XJ53D2HXIK3KRCUVYNC6GYEOC2G4YWHKPM37LBV5VHEWI27OA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.629380693 (522h19m8.500463738s) [noise=true , address=145.79.227.35:4483   , piece_id=MHMRYGHCAKCAYQX72SBC66KXJJ6XYO7I7AIXQKICZCG27A5BUCRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.629823797 (522h19m8.500906832s) [noise=true , address=45.91.116.7:4009     , piece_id=AAS4TYSEJ34Y5Y5VBC4GYOR3SXHV5YBIOCIQGTDCDSIESN3ZIYSA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.632972714 (522h19m8.504055749s) [noise=true , address=45.91.116.5:4300     , piece_id=H3SYBCRQGWYOYZFOONWYMPASTFNGSGHOESWLQO66BAKC3FR7VVUA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.636258209 (522h19m8.507341254s) [noise=true , address=145.79.227.7:4294    , piece_id=KYOMGQ6WD4OBYMJFU3U3SIPIHXT7KQCE52TSO2PW3XH7SMO66GWA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.638407703 (522h19m8.509490748s) [noise=true , address=45.91.116.6:4975     , piece_id=DBUCWDFWWBHVMXGEQDVUIWC7SDKVIEXWX2MRGXV2AU5JEYBAFKEA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.639644878 (522h19m8.510727923s) [noise=true , address=77.90.31.16:4735     , piece_id=PSQOYPWSM7HYUJORZKAA3JUJX5MDCO2CPUCXBPISMW6KECJ2X7DA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.640974219 (522h19m8.512057264s) [noise=true , address=145.79.227.47:4265   , piece_id=6GBIQLFWCFRE3TI4BLKXFC55DILA73D3AH6TJFJLDLE2HIYZSG7Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.642014540 (522h19m8.513097585s) [noise=true , address=77.90.31.18:4805     , piece_id=AV2GGH3SIEKBKX4Y2ZC73JK4CWOVYQZ4VFN4VOO2DHG44K34RIUA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:08.642466180 (522h19m8.513549215s) [noise=true , address=45.91.116.14:4190    , piece_id=BU6J5226JU7MPBQ66T2TFQTQSMNFZGA2BBNS6SIYFUALGBOQNYNQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 27 09:48:10.350298146 (522h19m10.221381181s) [upload=458210]: Next returned nil segment
Dec 27 09:48:10.350320840 (522h19m10.221403875s) [upload=458210]: Waiting for error group managing segments...
Dec 27 09:48:10.637362149 (522h19m10.508445194s) [noise=true , address=145.79.227.23:4546   , piece_id=5I5NAETOGAE2HJL2QUZEOUNFEOQTEZNWUXX3POFPGUM4RB27EMDQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.641386533 (522h19m10.512469578s) [noise=true , address=145.79.227.7:4294    , piece_id=KYOMGQ6WD4OBYMJFU3U3SIPIHXT7KQCE52TSO2PW3XH7SMO66GWA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.646472958 (522h19m10.517555993s) [noise=true , address=145.79.227.25:4617   , piece_id=IKEKZW3RXK7FLT554SBL7VVYUIKAN7NMRCD2ZMEK4MG7MMOBBRRA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.649878433 (522h19m10.520961488s) [noise=true , address=145.79.227.52:4447   , piece_id=RG4HQK2NVM7OFJI7ENCCVIT3UVPV6JH3H7ZPJJOGYIRFJMSUFUJA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.655144280 (522h19m10.526227325s) [noise=true , address=145.79.227.38:4978   , piece_id=EHERKEMYPM3KZFIEGIFVUVHDYQ55IOFAZCLBWO7AB34ZFVINGZLA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.658559583 (522h19m10.529642618s) [noise=true , address=145.79.227.14:4840   , piece_id=5C7XJ53D2HXIK3KRCUVYNC6GYEOC2G4YWHKPM37LBV5VHEWI27OA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.659443256 (522h19m10.530526301s) [noise=true , address=145.79.227.33:4871   , piece_id=MAYAQZK7SFMPEBARABRWQFH54ERV6XMFTBFN6LHOZNUI3Z42ROMQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.659925906 (522h19m10.531008941s) [noise=true , address=145.79.227.47:4265   , piece_id=6GBIQLFWCFRE3TI4BLKXFC55DILA73D3AH6TJFJLDLE2HIYZSG7Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.662302310 (522h19m10.533385355s) [noise=true , address=145.79.227.35:4483   , piece_id=MHMRYGHCAKCAYQX72SBC66KXJJ6XYO7I7AIXQKICZCG27A5BUCRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.833673052 (522h19m10.704756087s) [noise=true , address=77.90.31.18:4805     , piece_id=AV2GGH3SIEKBKX4Y2ZC73JK4CWOVYQZ4VFN4VOO2DHG44K34RIUA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.918881775 (522h19m10.78996482s) [noise=true , address=45.91.116.6:4975     , piece_id=DBUCWDFWWBHVMXGEQDVUIWC7SDKVIEXWX2MRGXV2AU5JEYBAFKEA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:10.956436055 (522h19m10.82751909s) [noise=true , address=77.90.31.16:4735     , piece_id=PSQOYPWSM7HYUJORZKAA3JUJX5MDCO2CPUCXBPISMW6KECJ2X7DA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:11.001264808 (522h19m10.872347873s) [noise=true , address=45.91.116.7:4009     , piece_id=AAS4TYSEJ34Y5Y5VBC4GYOR3SXHV5YBIOCIQGTDCDSIESN3ZIYSA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 27 09:48:11.001291398 (522h19m10.872374443s) [seg_pos={0 0}, upload=458210]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:48:11.001640813 (522h19m10.872723898s) [noise=true , address=45.91.116.5:4300     , piece_id=H3SYBCRQGWYOYZFOONWYMPASTFNGSGHOESWLQO66BAKC3FR7VVUA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: read tcp 188.166.225.170:54230->45.91.116.5:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 09:48:11.001651705 (522h19m10.87273474s) [noise=true , address=45.91.116.5:4300     , piece_id=H3SYBCRQGWYOYZFOONWYMPASTFNGSGHOESWLQO66BAKC3FR7VVUA, seg_pos={0 0}, upload=458210]: Long tail cancellation due to optimal threshold
Dec 27 09:48:11.001696299 (522h19m10.872779334s) [noise=true , address=45.91.116.14:4190    , piece_id=BU6J5226JU7MPBQ66T2TFQTQSMNFZGA2BBNS6SIYFUALGBOQNYNQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 27 09:48:11.001728110 (522h19m10.872811145s) [noise=true , address=45.91.116.14:4190    , piece_id=BU6J5226JU7MPBQ66T2TFQTQSMNFZGA2BBNS6SIYFUALGBOQNYNQ, seg_pos={0 0}, upload=458210]: Long tail cancellation due to optimal threshold
Dec 27 09:48:11.002158429 (522h19m10.873241474s) [noise=true , address=77.90.31.21:4877     , piece_id=W6MBKMNQKPS5B6D26D3QND6NBNC3JPRZWNK636DBU2LD3O2EX4VQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44902->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:44902->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44902->77.90.31.21:4877: use of closed network connection
Dec 27 09:48:11.002166544 (522h19m10.873249589s) [noise=true , address=77.90.31.21:4877     , piece_id=W6MBKMNQKPS5B6D26D3QND6NBNC3JPRZWNK636DBU2LD3O2EX4VQ, seg_pos={0 0}, upload=458210]: Long tail cancellation due to optimal threshold
Dec 27 09:48:11.002229314 (522h19m10.873312359s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:48:11.002334935 (522h19m10.87341797s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:48:11.002339854 (522h19m10.873422889s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:48:11.293664220 (522h19m11.164747286s) [upload=458210]: Done uploading segments...
Dec 27 09:48:11.293687624 (522h19m11.164770669s) [upload=458210]: Upload finished. err: <nil>
Duration: 30.01135459s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...............X........X.............X.....................]
duration: 
45.193358309s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:48:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 45.193377344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [.....X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..X]
duration: 
30.002181774s
trace: 
https://jaeger.dev.storj.io/trace/4389f8a9cecc82ec
FAIL (2025-12-27 09:48:05 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: IYTM6JMYEXB3Y25SZRGX7GBKOVD3XP6SWJ3LLBZQYOTZPDWGO4NQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: BA3SXHPP25AHMH3US7EEFNDKTZ2662PMEY4ABPCQUA5XI43RMXPQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: OVBJMCA5LGDYINS2OPPGALRBLYE7EFSADAU3BFIKF6FKLOFRPZOA) context deadline exceeded
======LOGS======
Dec 27 09:47:36.594593040 (522h18m36.465676085s) [upload=458201]: Starting upload
Dec 27 09:47:36.594646061 (522h18m36.465729096s) [upload=458201]: Done starting upload
Dec 27 09:47:36.594817698 (522h18m36.465900733s) [upload=458201]: Uploading segments...
Dec 27 09:47:36.594956332 (522h18m36.466039367s) [upload=458201]: Got next segment. Inline: false
Dec 27 09:47:36.595007920 (522h18m36.466090955s) [upload=458201]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:47:36.595016517 (522h18m36.466099592s) [upload=458201]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:47:36.838065496 (522h18m36.709148531s) [seg_pos={0 0}, upload=458201]: Begin upload segment...
Dec 27 09:47:36.838120531 (522h18m36.709203566s) [seg_pos={0 0}, upload=458201]: Done begin upload segment.
Dec 27 09:47:36.838200283 (522h18m36.709283318s) [noise=true , address=145.79.227.52:4446   , piece_id=R62ABJOOO5M4CMZQ7V6QFFTTKULZJVA5ZBAWT7KMBE4GCOA2ZKOQ, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.838205172 (522h18m36.709288207s) [noise=true , address=145.79.227.49:4339   , piece_id=KURCO4ZWRBDCYTEBBP3K4MMGPAA5SZIJ5XOP2TXU3AZO26N42E7A, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.838947426 (522h18m36.710030461s) [noise=true , address=77.90.31.14:4663     , piece_id=B43GLOOEY5EMZRJ7TBD6Z3QNBJ6K6N7QFOEBCOMTMUMYXIMXKAOA, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.839549312 (522h18m36.710632347s) [noise=true , address=145.79.227.7:4295    , piece_id=HQAGMAX452UOT4CXBN36UTSETQU5X76HZJSXK2DCUBDBHPAEI74A, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.839570292 (522h18m36.710653327s) [noise=true , address=45.91.116.8:4046     , piece_id=OVBJMCA5LGDYINS2OPPGALRBLYE7EFSADAU3BFIKF6FKLOFRPZOA, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.839928814 (522h18m36.71101186s) [noise=true , address=145.79.227.34:4906   , piece_id=RMNWF6TC6MIUEOWZSAKXRTNHBHBBJXWAYH43P7A7OWHGDEQEIOQA, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.840008286 (522h18m36.711091321s) [noise=true , address=77.90.31.21:4878     , piece_id=7VRLVYLZ5MLFMEBZI5AKHWX3UOVTD34FXDPC422S4WZMFPKTK24A, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.840334638 (522h18m36.711417673s) [noise=true , address=45.91.116.9:4262     , piece_id=WXMBGPWNVVU7VE7R423P436ILCXIF6KQ4OCAHNJ5HATD5ECNID2A, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.840414559 (522h18m36.711497595s) [noise=true , address=145.79.227.39:4014   , piece_id=Y2GA4XUA6JWM3XSKTQP3T6436MWLZS3WHAVN6VXLKQA6FOYVWWNQ, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.840694773 (522h18m36.711777818s) [noise=true , address=145.79.227.45:4229   , piece_id=EGSRFN36KFCZS664JSAIFOGG44F4HTL7Q4Q72K4BEB75KOKJZFBQ, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.841120073 (522h18m36.712203108s) [noise=true , address=145.79.227.30:4761   , piece_id=QT2GTGRGVDJIXXZZS67G26P4XRPT2VFYUJWNEKKM5TV46YEO6WUQ, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.841579808 (522h18m36.712662843s) [noise=true , address=45.91.116.12:4226    , piece_id=3A7NQXQAQAD3NYHDWF3EVF4TAMS5RVYXQVCEAC2VM7JIHXIXKSHQ, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.842929989 (522h18m36.714013034s) [noise=true , address=45.91.116.11:4120    , piece_id=JCVO3EJEFXCWU6GSQUKBN4MVAMSAT64KICGINTDJMZUOULCAGPWA, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.843675649 (522h18m36.714758715s) [noise=true , address=45.91.116.14:4189    , piece_id=IYTM6JMYEXB3Y25SZRGX7GBKOVD3XP6SWJ3LLBZQYOTZPDWGO4NQ, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.845166187 (522h18m36.716249232s) [noise=true , address=45.91.116.7:4012     , piece_id=BA3SXHPP25AHMH3US7EEFNDKTZ2662PMEY4ABPCQUA5XI43RMXPQ, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:36.846876685 (522h18m36.71795973s) [noise=true , address=145.79.227.46:4601   , piece_id=YXUU6WQ2E4PAIVI2XDXGADVGV7YIRCY754DFXAJTQBT5C725ZMJQ, seg_pos={0 0}, upload=458201]: Uploading piece...
Dec 27 09:47:38.556770210 (522h18m38.427853245s) [upload=458201]: Next returned nil segment
Dec 27 09:47:38.556777092 (522h18m38.427860138s) [upload=458201]: Waiting for error group managing segments...
Dec 27 09:47:38.844802797 (522h18m38.715885832s) [noise=true , address=145.79.227.45:4229   , piece_id=EGSRFN36KFCZS664JSAIFOGG44F4HTL7Q4Q72K4BEB75KOKJZFBQ, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:38.851893189 (522h18m38.722976224s) [noise=true , address=145.79.227.46:4601   , piece_id=YXUU6WQ2E4PAIVI2XDXGADVGV7YIRCY754DFXAJTQBT5C725ZMJQ, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:38.852850351 (522h18m38.723933396s) [noise=true , address=145.79.227.52:4446   , piece_id=R62ABJOOO5M4CMZQ7V6QFFTTKULZJVA5ZBAWT7KMBE4GCOA2ZKOQ, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:38.866637745 (522h18m38.73772078s) [noise=true , address=145.79.227.39:4014   , piece_id=Y2GA4XUA6JWM3XSKTQP3T6436MWLZS3WHAVN6VXLKQA6FOYVWWNQ, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:38.875535748 (522h18m38.746618783s) [noise=true , address=145.79.227.49:4339   , piece_id=KURCO4ZWRBDCYTEBBP3K4MMGPAA5SZIJ5XOP2TXU3AZO26N42E7A, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:38.877102812 (522h18m38.748185857s) [noise=true , address=145.79.227.7:4295    , piece_id=HQAGMAX452UOT4CXBN36UTSETQU5X76HZJSXK2DCUBDBHPAEI74A, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:39.073441287 (522h18m38.944524332s) [noise=true , address=45.91.116.12:4226    , piece_id=3A7NQXQAQAD3NYHDWF3EVF4TAMS5RVYXQVCEAC2VM7JIHXIXKSHQ, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:39.084660390 (522h18m38.955743425s) [noise=true , address=45.91.116.11:4120    , piece_id=JCVO3EJEFXCWU6GSQUKBN4MVAMSAT64KICGINTDJMZUOULCAGPWA, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:39.100124818 (522h18m38.971207863s) [noise=true , address=77.90.31.21:4878     , piece_id=7VRLVYLZ5MLFMEBZI5AKHWX3UOVTD34FXDPC422S4WZMFPKTK24A, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:39.210637845 (522h18m39.08172088s) [noise=true , address=45.91.116.9:4262     , piece_id=WXMBGPWNVVU7VE7R423P436ILCXIF6KQ4OCAHNJ5HATD5ECNID2A, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:39.232453065 (522h18m39.1035361s) [noise=true , address=45.91.116.8:4046     , piece_id=OVBJMCA5LGDYINS2OPPGALRBLYE7EFSADAU3BFIKF6FKLOFRPZOA, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:39.524977867 (522h18m39.396060902s) [noise=true , address=45.91.116.14:4189    , piece_id=IYTM6JMYEXB3Y25SZRGX7GBKOVD3XP6SWJ3LLBZQYOTZPDWGO4NQ, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:39.826992457 (522h18m39.698075492s) [noise=true , address=45.91.116.7:4012     , piece_id=BA3SXHPP25AHMH3US7EEFNDKTZ2662PMEY4ABPCQUA5XI43RMXPQ, seg_pos={0 0}, upload=458201]: Done uploading piece. err: <nil>
Dec 27 09:47:39.827016262 (522h18m39.698099297s) [seg_pos={0 0}, upload=458201]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:47:39.827221814 (522h18m39.698304859s) [noise=true , address=145.79.227.30:4761   , piece_id=QT2GTGRGVDJIXXZZS67G26P4XRPT2VFYUJWNEKKM5TV46YEO6WUQ, seg_pos={0 0}, upload=458201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 09:47:39.827230270 (522h18m39.698313315s) [noise=true , address=145.79.227.30:4761   , piece_id=QT2GTGRGVDJIXXZZS67G26P4XRPT2VFYUJWNEKKM5TV46YEO6WUQ, seg_pos={0 0}, upload=458201]: Long tail cancellation due to optimal threshold
Dec 27 09:47:39.827257361 (522h18m39.698340406s) [noise=true , address=145.79.227.34:4906   , piece_id=RMNWF6TC6MIUEOWZSAKXRTNHBHBBJXWAYH43P7A7OWHGDEQEIOQA, seg_pos={0 0}, upload=458201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 27 09:47:39.827268903 (522h18m39.698351938s) [noise=true , address=145.79.227.34:4906   , piece_id=RMNWF6TC6MIUEOWZSAKXRTNHBHBBJXWAYH43P7A7OWHGDEQEIOQA, seg_pos={0 0}, upload=458201]: Long tail cancellation due to optimal threshold
Dec 27 09:47:39.827382731 (522h18m39.698465776s) [noise=true , address=77.90.31.14:4663     , piece_id=B43GLOOEY5EMZRJ7TBD6Z3QNBJ6K6N7QFOEBCOMTMUMYXIMXKAOA, seg_pos={0 0}, upload=458201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 27 09:47:39.827390866 (522h18m39.698473911s) [noise=true , address=77.90.31.14:4663     , piece_id=B43GLOOEY5EMZRJ7TBD6Z3QNBJ6K6N7QFOEBCOMTMUMYXIMXKAOA, seg_pos={0 0}, upload=458201]: Long tail cancellation due to optimal threshold
Dec 27 09:47:39.827437354 (522h18m39.698520389s) [seg_pos={0 0}, upload=458201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:47:39.827514280 (522h18m39.698597315s) [upload=458201]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:47:39.827520844 (522h18m39.698603879s) [upload=458201]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:47:40.120823626 (522h18m39.991906671s) [upload=458201]: Done uploading segments...
Dec 27 09:47:40.120848142 (522h18m39.991931187s) [upload=458201]: Upload finished. err: <nil>
Duration: 30.00222207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [................X........X.............X....................]
duration: 
46.099021705s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:47:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 46.099042474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [......X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX..]
duration: 
6.042335569s
trace: 
https://jaeger.dev.storj.io/trace/22a9c06e07f1f83f
FAIL (2025-12-27 09:47:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ) context deadline exceeded
======LOGS======
 09:45:55.980926018 (522h16m55.852009053s) [upload=458177]: Starting upload
Dec 27 09:45:55.980990791 (522h16m55.852073826s) [upload=458177]: Done starting upload
Dec 27 09:45:55.981060945 (522h16m55.85214399s) [upload=458177]: Uploading segments...
Dec 27 09:45:55.981306222 (522h16m55.852389267s) [upload=458177]: Got next segment. Inline: false
Dec 27 09:45:55.981359173 (522h16m55.852442218s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:45:55.981391324 (522h16m55.852474369s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:45:56.223617668 (522h16m56.094700713s) [seg_pos={0 0}, upload=458177]: Begin upload segment...
Dec 27 09:45:56.223680407 (522h16m56.094763452s) [seg_pos={0 0}, upload=458177]: Done begin upload segment.
Dec 27 09:45:56.223718470 (522h16m56.094801515s) [noise=true , address=145.79.227.7:4294    , piece_id=D3IFSRLBM43DSVEBR3ZJCKLKLJUFW6EJOX4AN4BGZKWJRHLWOEUQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.223791839 (522h16m56.094874895s) [noise=true , address=145.79.227.17:4971   , piece_id=AE57BZFDH5EVKL5VHNR5PAZ7HH7XWNVKRX2XSGEOFPQIHVUHCD7Q, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.224645836 (522h16m56.095728891s) [noise=true , address=145.79.227.48:4303   , piece_id=KEPD7373D3VME7T6XKOFBIF5A5KQIQXFLHXNACX73ADKH2GO7VPQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.224780141 (522h16m56.095863186s) [noise=true , address=145.79.227.43:4159   , piece_id=BCMGCDFOO7YREMA23427BCULRGGR6RL3AWOKVSH27SIOBNNYGGGA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225084461 (522h16m56.096167496s) [noise=true , address=145.79.227.28:4727   , piece_id=I23BSIEN2PYYE6MYYMN5UP6XYU4DNHY6MKGVT26NNVB5YOZA2TFQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225374794 (522h16m56.096457829s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225666559 (522h16m56.096749594s) [noise=true , address=45.91.116.5:4299     , piece_id=6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226175098 (522h16m56.097258133s) [noise=true , address=145.79.227.36:4519   , piece_id=MLSA3RQIKBD52PU2MIPYOU5RVJISMCXSEBJVNBTW6JZFXO5JBECQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226501960 (522h16m56.097584995s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226858659 (522h16m56.097941704s) [noise=true , address=145.79.227.11:4729   , piece_id=PTFJBSTK7LFWPY6QCOGA4RSJWMYCRRBDXTECW272ZI2JZQCBRERQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.227455707 (522h16m56.098538742s) [noise=true , address=145.79.227.38:4979   , piece_id=IEAXGOIFLXUYZP3NVM3ZIUPRUUUVSRYL23QA4UYALPZJF2R6JSSQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.227962221 (522h16m56.099045256s) [noise=true , address=145.79.227.47:4266   , piece_id=S7KRXNSPWFFSZY3PL63TYMCNS33MTVHQG7YNECUA74FJFSV424VA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.228304964 (522h16m56.099387999s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.228753317 (522h16m56.099836352s) [noise=true , address=77.90.31.12:4589     , piece_id=E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.229278988 (522h16m56.100362023s) [noise=true , address=145.79.227.25:4618   , piece_id=DISONM67D7MSA65M4RQTYM2ZDQHBPA67OLRIUNPRUHKPFMS3U3YQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.230289603 (522h16m56.101372638s) [noise=true , address=145.79.227.2:4114    , piece_id=2QHMKY3DFTVG5UYPLPNDMQ6JFULORKDBO3R2O225YQTGD3RKLH5A, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:57.934831154 (522h16m57.805914189s) [upload=458177]: Next returned nil segment
Dec 27 09:45:57.934861442 (522h16m57.805944477s) [upload=458177]: Waiting for error group managing segments...
Dec 27 09:45:58.224365110 (522h16m58.095448155s) [noise=true , address=145.79.227.36:4519   , piece_id=MLSA3RQIKBD52PU2MIPYOU5RVJISMCXSEBJVNBTW6JZFXO5JBECQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.227262438 (522h16m58.098345483s) [noise=true , address=145.79.227.7:4294    , piece_id=D3IFSRLBM43DSVEBR3ZJCKLKLJUFW6EJOX4AN4BGZKWJRHLWOEUQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.232874534 (522h16m58.103957569s) [noise=true , address=145.79.227.38:4979   , piece_id=IEAXGOIFLXUYZP3NVM3ZIUPRUUUVSRYL23QA4UYALPZJF2R6JSSQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.233717950 (522h16m58.104800995s) [noise=true , address=145.79.227.43:4159   , piece_id=BCMGCDFOO7YREMA23427BCULRGGR6RL3AWOKVSH27SIOBNNYGGGA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.234412943 (522h16m58.105495988s) [noise=true , address=145.79.227.11:4729   , piece_id=PTFJBSTK7LFWPY6QCOGA4RSJWMYCRRBDXTECW272ZI2JZQCBRERQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.240296837 (522h16m58.111379882s) [noise=true , address=145.79.227.25:4618   , piece_id=DISONM67D7MSA65M4RQTYM2ZDQHBPA67OLRIUNPRUHKPFMS3U3YQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.248283536 (522h16m58.119366571s) [noise=true , address=145.79.227.28:4727   , piece_id=I23BSIEN2PYYE6MYYMN5UP6XYU4DNHY6MKGVT26NNVB5YOZA2TFQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.254312225 (522h16m58.12539526s) [noise=true , address=145.79.227.48:4303   , piece_id=KEPD7373D3VME7T6XKOFBIF5A5KQIQXFLHXNACX73ADKH2GO7VPQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.255439121 (522h16m58.126522156s) [noise=true , address=145.79.227.17:4971   , piece_id=AE57BZFDH5EVKL5VHNR5PAZ7HH7XWNVKRX2XSGEOFPQIHVUHCD7Q, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.257083743 (522h16m58.128166778s) [noise=true , address=145.79.227.47:4266   , piece_id=S7KRXNSPWFFSZY3PL63TYMCNS33MTVHQG7YNECUA74FJFSV424VA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.257764940 (522h16m58.128847975s) [noise=true , address=145.79.227.2:4114    , piece_id=2QHMKY3DFTVG5UYPLPNDMQ6JFULORKDBO3R2O225YQTGD3RKLH5A, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.464352861 (522h16m58.335435906s) [noise=true , address=77.90.31.12:4589     , piece_id=E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.514814509 (522h16m58.385897544s) [noise=true , address=45.91.116.5:4299     , piece_id=6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.514833245 (522h16m58.385916281s) [seg_pos={0 0}, upload=458177]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:45:58.515006996 (522h16m58.386090031s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection
Dec 27 09:45:58.515012907 (522h16m58.386095942s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515266380 (522h16m58.386349415s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF
Dec 27 09:45:58.515272662 (522h16m58.386355707s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515314372 (522h16m58.386397407s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:45:58.515346253 (522h16m58.386429298s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515375968 (522h16m58.386459003s) [seg_pos={0 0}, upload=458177]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:45:58.515505676 (522h16m58.386588721s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:45:58.515512599 (522h16m58.386595634s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:45:58.809854019 (522h16m58.680937064s) [upload=458177]: Done uploading segments...
Dec 27 09:45:58.809877414 (522h16m58.680960449s) [upload=458177]: Upload finished. err: <nil>
Duration: 6.042360076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.................X........X.............X...................]
duration: 
55.873701461s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:46:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 55.873732821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [.......X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX.]
duration: 
5.663683298s
trace: 
https://jaeger.dev.storj.io/trace/22a9c06e07f1f83f
FAIL (2025-12-27 09:46:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ) context deadline exceeded
======LOGS======
 09:45:55.980926018 (522h16m55.852009053s) [upload=458177]: Starting upload
Dec 27 09:45:55.980990791 (522h16m55.852073826s) [upload=458177]: Done starting upload
Dec 27 09:45:55.981060945 (522h16m55.85214399s) [upload=458177]: Uploading segments...
Dec 27 09:45:55.981306222 (522h16m55.852389267s) [upload=458177]: Got next segment. Inline: false
Dec 27 09:45:55.981359173 (522h16m55.852442218s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:45:55.981391324 (522h16m55.852474369s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:45:56.223617668 (522h16m56.094700713s) [seg_pos={0 0}, upload=458177]: Begin upload segment...
Dec 27 09:45:56.223680407 (522h16m56.094763452s) [seg_pos={0 0}, upload=458177]: Done begin upload segment.
Dec 27 09:45:56.223718470 (522h16m56.094801515s) [noise=true , address=145.79.227.7:4294    , piece_id=D3IFSRLBM43DSVEBR3ZJCKLKLJUFW6EJOX4AN4BGZKWJRHLWOEUQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.223791839 (522h16m56.094874895s) [noise=true , address=145.79.227.17:4971   , piece_id=AE57BZFDH5EVKL5VHNR5PAZ7HH7XWNVKRX2XSGEOFPQIHVUHCD7Q, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.224645836 (522h16m56.095728891s) [noise=true , address=145.79.227.48:4303   , piece_id=KEPD7373D3VME7T6XKOFBIF5A5KQIQXFLHXNACX73ADKH2GO7VPQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.224780141 (522h16m56.095863186s) [noise=true , address=145.79.227.43:4159   , piece_id=BCMGCDFOO7YREMA23427BCULRGGR6RL3AWOKVSH27SIOBNNYGGGA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225084461 (522h16m56.096167496s) [noise=true , address=145.79.227.28:4727   , piece_id=I23BSIEN2PYYE6MYYMN5UP6XYU4DNHY6MKGVT26NNVB5YOZA2TFQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225374794 (522h16m56.096457829s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225666559 (522h16m56.096749594s) [noise=true , address=45.91.116.5:4299     , piece_id=6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226175098 (522h16m56.097258133s) [noise=true , address=145.79.227.36:4519   , piece_id=MLSA3RQIKBD52PU2MIPYOU5RVJISMCXSEBJVNBTW6JZFXO5JBECQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226501960 (522h16m56.097584995s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226858659 (522h16m56.097941704s) [noise=true , address=145.79.227.11:4729   , piece_id=PTFJBSTK7LFWPY6QCOGA4RSJWMYCRRBDXTECW272ZI2JZQCBRERQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.227455707 (522h16m56.098538742s) [noise=true , address=145.79.227.38:4979   , piece_id=IEAXGOIFLXUYZP3NVM3ZIUPRUUUVSRYL23QA4UYALPZJF2R6JSSQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.227962221 (522h16m56.099045256s) [noise=true , address=145.79.227.47:4266   , piece_id=S7KRXNSPWFFSZY3PL63TYMCNS33MTVHQG7YNECUA74FJFSV424VA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.228304964 (522h16m56.099387999s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.228753317 (522h16m56.099836352s) [noise=true , address=77.90.31.12:4589     , piece_id=E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.229278988 (522h16m56.100362023s) [noise=true , address=145.79.227.25:4618   , piece_id=DISONM67D7MSA65M4RQTYM2ZDQHBPA67OLRIUNPRUHKPFMS3U3YQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.230289603 (522h16m56.101372638s) [noise=true , address=145.79.227.2:4114    , piece_id=2QHMKY3DFTVG5UYPLPNDMQ6JFULORKDBO3R2O225YQTGD3RKLH5A, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:57.934831154 (522h16m57.805914189s) [upload=458177]: Next returned nil segment
Dec 27 09:45:57.934861442 (522h16m57.805944477s) [upload=458177]: Waiting for error group managing segments...
Dec 27 09:45:58.224365110 (522h16m58.095448155s) [noise=true , address=145.79.227.36:4519   , piece_id=MLSA3RQIKBD52PU2MIPYOU5RVJISMCXSEBJVNBTW6JZFXO5JBECQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.227262438 (522h16m58.098345483s) [noise=true , address=145.79.227.7:4294    , piece_id=D3IFSRLBM43DSVEBR3ZJCKLKLJUFW6EJOX4AN4BGZKWJRHLWOEUQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.232874534 (522h16m58.103957569s) [noise=true , address=145.79.227.38:4979   , piece_id=IEAXGOIFLXUYZP3NVM3ZIUPRUUUVSRYL23QA4UYALPZJF2R6JSSQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.233717950 (522h16m58.104800995s) [noise=true , address=145.79.227.43:4159   , piece_id=BCMGCDFOO7YREMA23427BCULRGGR6RL3AWOKVSH27SIOBNNYGGGA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.234412943 (522h16m58.105495988s) [noise=true , address=145.79.227.11:4729   , piece_id=PTFJBSTK7LFWPY6QCOGA4RSJWMYCRRBDXTECW272ZI2JZQCBRERQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.240296837 (522h16m58.111379882s) [noise=true , address=145.79.227.25:4618   , piece_id=DISONM67D7MSA65M4RQTYM2ZDQHBPA67OLRIUNPRUHKPFMS3U3YQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.248283536 (522h16m58.119366571s) [noise=true , address=145.79.227.28:4727   , piece_id=I23BSIEN2PYYE6MYYMN5UP6XYU4DNHY6MKGVT26NNVB5YOZA2TFQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.254312225 (522h16m58.12539526s) [noise=true , address=145.79.227.48:4303   , piece_id=KEPD7373D3VME7T6XKOFBIF5A5KQIQXFLHXNACX73ADKH2GO7VPQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.255439121 (522h16m58.126522156s) [noise=true , address=145.79.227.17:4971   , piece_id=AE57BZFDH5EVKL5VHNR5PAZ7HH7XWNVKRX2XSGEOFPQIHVUHCD7Q, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.257083743 (522h16m58.128166778s) [noise=true , address=145.79.227.47:4266   , piece_id=S7KRXNSPWFFSZY3PL63TYMCNS33MTVHQG7YNECUA74FJFSV424VA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.257764940 (522h16m58.128847975s) [noise=true , address=145.79.227.2:4114    , piece_id=2QHMKY3DFTVG5UYPLPNDMQ6JFULORKDBO3R2O225YQTGD3RKLH5A, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.464352861 (522h16m58.335435906s) [noise=true , address=77.90.31.12:4589     , piece_id=E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.514814509 (522h16m58.385897544s) [noise=true , address=45.91.116.5:4299     , piece_id=6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.514833245 (522h16m58.385916281s) [seg_pos={0 0}, upload=458177]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:45:58.515006996 (522h16m58.386090031s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection
Dec 27 09:45:58.515012907 (522h16m58.386095942s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515266380 (522h16m58.386349415s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF
Dec 27 09:45:58.515272662 (522h16m58.386355707s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515314372 (522h16m58.386397407s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:45:58.515346253 (522h16m58.386429298s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515375968 (522h16m58.386459003s) [seg_pos={0 0}, upload=458177]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:45:58.515505676 (522h16m58.386588721s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:45:58.515512599 (522h16m58.386595634s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:45:58.809854019 (522h16m58.680937064s) [upload=458177]: Done uploading segments...
Dec 27 09:45:58.809877414 (522h16m58.680960449s) [upload=458177]: Upload finished. err: <nil>
Duration: 5.663697576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [........X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..XX]
duration: 
30.003688463s
trace: 
https://jaeger.dev.storj.io/trace/22a9c06e07f1f83f
FAIL (2025-12-27 09:46:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ) context deadline exceeded
======LOGS======
 09:45:55.980926018 (522h16m55.852009053s) [upload=458177]: Starting upload
Dec 27 09:45:55.980990791 (522h16m55.852073826s) [upload=458177]: Done starting upload
Dec 27 09:45:55.981060945 (522h16m55.85214399s) [upload=458177]: Uploading segments...
Dec 27 09:45:55.981306222 (522h16m55.852389267s) [upload=458177]: Got next segment. Inline: false
Dec 27 09:45:55.981359173 (522h16m55.852442218s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:45:55.981391324 (522h16m55.852474369s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:45:56.223617668 (522h16m56.094700713s) [seg_pos={0 0}, upload=458177]: Begin upload segment...
Dec 27 09:45:56.223680407 (522h16m56.094763452s) [seg_pos={0 0}, upload=458177]: Done begin upload segment.
Dec 27 09:45:56.223718470 (522h16m56.094801515s) [noise=true , address=145.79.227.7:4294    , piece_id=D3IFSRLBM43DSVEBR3ZJCKLKLJUFW6EJOX4AN4BGZKWJRHLWOEUQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.223791839 (522h16m56.094874895s) [noise=true , address=145.79.227.17:4971   , piece_id=AE57BZFDH5EVKL5VHNR5PAZ7HH7XWNVKRX2XSGEOFPQIHVUHCD7Q, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.224645836 (522h16m56.095728891s) [noise=true , address=145.79.227.48:4303   , piece_id=KEPD7373D3VME7T6XKOFBIF5A5KQIQXFLHXNACX73ADKH2GO7VPQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.224780141 (522h16m56.095863186s) [noise=true , address=145.79.227.43:4159   , piece_id=BCMGCDFOO7YREMA23427BCULRGGR6RL3AWOKVSH27SIOBNNYGGGA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225084461 (522h16m56.096167496s) [noise=true , address=145.79.227.28:4727   , piece_id=I23BSIEN2PYYE6MYYMN5UP6XYU4DNHY6MKGVT26NNVB5YOZA2TFQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225374794 (522h16m56.096457829s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.225666559 (522h16m56.096749594s) [noise=true , address=45.91.116.5:4299     , piece_id=6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226175098 (522h16m56.097258133s) [noise=true , address=145.79.227.36:4519   , piece_id=MLSA3RQIKBD52PU2MIPYOU5RVJISMCXSEBJVNBTW6JZFXO5JBECQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226501960 (522h16m56.097584995s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.226858659 (522h16m56.097941704s) [noise=true , address=145.79.227.11:4729   , piece_id=PTFJBSTK7LFWPY6QCOGA4RSJWMYCRRBDXTECW272ZI2JZQCBRERQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.227455707 (522h16m56.098538742s) [noise=true , address=145.79.227.38:4979   , piece_id=IEAXGOIFLXUYZP3NVM3ZIUPRUUUVSRYL23QA4UYALPZJF2R6JSSQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.227962221 (522h16m56.099045256s) [noise=true , address=145.79.227.47:4266   , piece_id=S7KRXNSPWFFSZY3PL63TYMCNS33MTVHQG7YNECUA74FJFSV424VA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.228304964 (522h16m56.099387999s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.228753317 (522h16m56.099836352s) [noise=true , address=77.90.31.12:4589     , piece_id=E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.229278988 (522h16m56.100362023s) [noise=true , address=145.79.227.25:4618   , piece_id=DISONM67D7MSA65M4RQTYM2ZDQHBPA67OLRIUNPRUHKPFMS3U3YQ, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:56.230289603 (522h16m56.101372638s) [noise=true , address=145.79.227.2:4114    , piece_id=2QHMKY3DFTVG5UYPLPNDMQ6JFULORKDBO3R2O225YQTGD3RKLH5A, seg_pos={0 0}, upload=458177]: Uploading piece...
Dec 27 09:45:57.934831154 (522h16m57.805914189s) [upload=458177]: Next returned nil segment
Dec 27 09:45:57.934861442 (522h16m57.805944477s) [upload=458177]: Waiting for error group managing segments...
Dec 27 09:45:58.224365110 (522h16m58.095448155s) [noise=true , address=145.79.227.36:4519   , piece_id=MLSA3RQIKBD52PU2MIPYOU5RVJISMCXSEBJVNBTW6JZFXO5JBECQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.227262438 (522h16m58.098345483s) [noise=true , address=145.79.227.7:4294    , piece_id=D3IFSRLBM43DSVEBR3ZJCKLKLJUFW6EJOX4AN4BGZKWJRHLWOEUQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.232874534 (522h16m58.103957569s) [noise=true , address=145.79.227.38:4979   , piece_id=IEAXGOIFLXUYZP3NVM3ZIUPRUUUVSRYL23QA4UYALPZJF2R6JSSQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.233717950 (522h16m58.104800995s) [noise=true , address=145.79.227.43:4159   , piece_id=BCMGCDFOO7YREMA23427BCULRGGR6RL3AWOKVSH27SIOBNNYGGGA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.234412943 (522h16m58.105495988s) [noise=true , address=145.79.227.11:4729   , piece_id=PTFJBSTK7LFWPY6QCOGA4RSJWMYCRRBDXTECW272ZI2JZQCBRERQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.240296837 (522h16m58.111379882s) [noise=true , address=145.79.227.25:4618   , piece_id=DISONM67D7MSA65M4RQTYM2ZDQHBPA67OLRIUNPRUHKPFMS3U3YQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.248283536 (522h16m58.119366571s) [noise=true , address=145.79.227.28:4727   , piece_id=I23BSIEN2PYYE6MYYMN5UP6XYU4DNHY6MKGVT26NNVB5YOZA2TFQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.254312225 (522h16m58.12539526s) [noise=true , address=145.79.227.48:4303   , piece_id=KEPD7373D3VME7T6XKOFBIF5A5KQIQXFLHXNACX73ADKH2GO7VPQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.255439121 (522h16m58.126522156s) [noise=true , address=145.79.227.17:4971   , piece_id=AE57BZFDH5EVKL5VHNR5PAZ7HH7XWNVKRX2XSGEOFPQIHVUHCD7Q, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.257083743 (522h16m58.128166778s) [noise=true , address=145.79.227.47:4266   , piece_id=S7KRXNSPWFFSZY3PL63TYMCNS33MTVHQG7YNECUA74FJFSV424VA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.257764940 (522h16m58.128847975s) [noise=true , address=145.79.227.2:4114    , piece_id=2QHMKY3DFTVG5UYPLPNDMQ6JFULORKDBO3R2O225YQTGD3RKLH5A, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.464352861 (522h16m58.335435906s) [noise=true , address=77.90.31.12:4589     , piece_id=E4SL52MKTCRYZSKHRLT4QGICCS7BPO7NOOYV7J23Q2Q3YJT4O3KQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.514814509 (522h16m58.385897544s) [noise=true , address=45.91.116.5:4299     , piece_id=6BTKB42FM37EHNNYBJ4A5DZK327RK4LWYYSP74TDIBBTVRNM5RDQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: <nil>
Dec 27 09:45:58.514833245 (522h16m58.385916281s) [seg_pos={0 0}, upload=458177]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:45:58.515006996 (522h16m58.386090031s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58896->77.90.31.15:4700: use of closed network connection
Dec 27 09:45:58.515012907 (522h16m58.386095942s) [noise=true , address=77.90.31.15:4700     , piece_id=DAAGXC6FH5B3H3VCTLNGI42WNCIENLUFMCJJRSGEFBG4L24K4O7Q, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515266380 (522h16m58.386349415s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF
Dec 27 09:45:58.515272662 (522h16m58.386355707s) [noise=true , address=145.79.227.8:4330    , piece_id=7N3HBE7NI74NC2THA6FODN24RL3N7BU6ZKIAHTX5OYTKOHYDAHQQ, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515314372 (522h16m58.386397407s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 27 09:45:58.515346253 (522h16m58.386429298s) [noise=true , address=45.91.116.10:4084    , piece_id=K3RU23YZSYBO47NVHBG6NIVODXCKN65RRERG3MYAWD4IM43ZSIAA, seg_pos={0 0}, upload=458177]: Long tail cancellation due to optimal threshold
Dec 27 09:45:58.515375968 (522h16m58.386459003s) [seg_pos={0 0}, upload=458177]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:45:58.515505676 (522h16m58.386588721s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:45:58.515512599 (522h16m58.386595634s) [upload=458177]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:45:58.809854019 (522h16m58.680937064s) [upload=458177]: Done uploading segments...
Dec 27 09:45:58.809877414 (522h16m58.680960449s) [upload=458177]: Upload finished. err: <nil>
Duration: 30.003779727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [X........X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..X]
duration: 
30.002973838s
trace: 
https://jaeger.dev.storj.io/trace/420f0d77be819da
FAIL (2025-12-27 09:45:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: E4RZUUW55HGZXQLNUSVCYIWFXAZML7AG6T5KEV6KTNI6BIA7NY4A) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: GOBU6AFGRD2NSF3EK2NSVXH2PGFDH2XCC2YJCUUZNNHV25CN5QTQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: VJEBW52OX5ZC3Y66N6JZYZVZHDX2ZDYYGR62YREUEWCX22QM74FQ) context deadline exceeded
======LOGS======
 27 09:45:15.929169532 (522h16m15.800252567s) [upload=458165]: Starting upload
Dec 27 09:45:15.929227754 (522h16m15.800310789s) [upload=458165]: Done starting upload
Dec 27 09:45:15.929421692 (522h16m15.800504737s) [upload=458165]: Uploading segments...
Dec 27 09:45:15.929467731 (522h16m15.800550766s) [upload=458165]: Got next segment. Inline: false
Dec 27 09:45:15.929501234 (522h16m15.80058427s) [upload=458165]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:45:15.929506044 (522h16m15.800589079s) [upload=458165]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:45:16.171963198 (522h16m16.043046233s) [seg_pos={0 0}, upload=458165]: Begin upload segment...
Dec 27 09:45:16.172007161 (522h16m16.043090206s) [seg_pos={0 0}, upload=458165]: Done begin upload segment.
Dec 27 09:45:16.172034924 (522h16m16.043117969s) [noise=true , address=45.91.116.9:4262     , piece_id=GU2IXOE4O7PBYCXOBJDPPC5XLLTWEBPLFS26BGCY7OX2XXAI4QJQ, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.172099958 (522h16m16.043183003s) [noise=true , address=77.90.31.10:4520     , piece_id=GOBU6AFGRD2NSF3EK2NSVXH2PGFDH2XCC2YJCUUZNNHV25CN5QTQ, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.172632011 (522h16m16.043715046s) [noise=true , address=45.91.116.7:4011     , piece_id=VJEBW52OX5ZC3Y66N6JZYZVZHDX2ZDYYGR62YREUEWCX22QM74FQ, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.173111905 (522h16m16.04419494s) [noise=true , address=45.91.116.11:4119    , piece_id=UF75UBTJHR6I52T22SRS2IM72Z3CVIR7C2JFDBI57KEMVT7GBN5A, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.173581228 (522h16m16.044664263s) [noise=true , address=45.91.116.13:4156    , piece_id=NHOOPPW4CPTZSMU63KVDKAJSMQF5FAGNQZWJIN7GP5KRXHOWQ73Q, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.174170931 (522h16m16.045253966s) [noise=true , address=145.79.227.35:4481   , piece_id=RVPQJXHQOTZE2UAJDO6S37DRLKHUT52JMQ4ARPO2ER4MU2VLATBQ, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.174482595 (522h16m16.04556563s) [noise=true , address=145.79.227.10:4402   , piece_id=TZ7GDKKHWYDDPWYR6Z23XKCHL6MVRCTQGUNK4NYLTLRZZOJT4GAQ, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.174909337 (522h16m16.045992382s) [noise=true , address=145.79.227.25:4618   , piece_id=AIGHXCTF4VR44BQIV5SFNP5WNVB7UNZD2JGSDERGIIXZCJX57SEQ, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.175985617 (522h16m16.047068652s) [noise=true , address=145.79.227.47:4265   , piece_id=2C5A6NJTYAYDEGBJOYLMXMOOHRXR5HQEGZ4NLOOE5XJHY7IXRA6Q, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.176433650 (522h16m16.047516685s) [noise=true , address=45.91.116.10:4083    , piece_id=VYZ5LCPNNDKZGQ4E3DKPLMI7Y4EACVZIWUSSBXMZ2K2NKL36MGUA, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.177607956 (522h16m16.048690991s) [noise=true , address=145.79.227.29:4943   , piece_id=BACEAWMVPVEQL6OG23VOBQUDNYY4BQ53UKNBIJTU4HFV7CN45RAA, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.180247092 (522h16m16.051330127s) [noise=true , address=145.79.227.7:4294    , piece_id=EURHN3JZWUE3J3UCPTUCSIFY62FGMZ555CLLQV3DGREDY57AGPYQ, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.181513503 (522h16m16.052596538s) [noise=true , address=77.90.31.12:4589     , piece_id=PXERZ4ZIS6LIQ6ONVH5PW57JRKSWF36AIYHGTEUOVEPJXMZNGHNA, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.182845639 (522h16m16.053928684s) [noise=true , address=45.91.116.6:4973     , piece_id=E4RZUUW55HGZXQLNUSVCYIWFXAZML7AG6T5KEV6KTNI6BIA7NY4A, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.184115388 (522h16m16.055198433s) [noise=true , address=45.91.116.14:4191    , piece_id=RXCUMIEZGQRTILRMSVYH3QH3HF46XJV2RZXMJXSU7ZXZVEGCDRVA, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:16.185414481 (522h16m16.056497516s) [noise=true , address=77.90.31.21:4877     , piece_id=7EC3QLVJEE3XTGIYOZUYCZ5VOMEFEIMMGSDOORJSWXAMU42EXFQA, seg_pos={0 0}, upload=458165]: Uploading piece...
Dec 27 09:45:17.887823220 (522h16m17.758906255s) [upload=458165]: Next returned nil segment
Dec 27 09:45:17.887849129 (522h16m17.758932174s) [upload=458165]: Waiting for error group managing segments...
Dec 27 09:45:18.170470263 (522h16m18.041553318s) [noise=true , address=145.79.227.10:4402   , piece_id=TZ7GDKKHWYDDPWYR6Z23XKCHL6MVRCTQGUNK4NYLTLRZZOJT4GAQ, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.173782890 (522h16m18.044865935s) [noise=true , address=145.79.227.35:4481   , piece_id=RVPQJXHQOTZE2UAJDO6S37DRLKHUT52JMQ4ARPO2ER4MU2VLATBQ, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.195052191 (522h16m18.066135236s) [noise=true , address=145.79.227.29:4943   , piece_id=BACEAWMVPVEQL6OG23VOBQUDNYY4BQ53UKNBIJTU4HFV7CN45RAA, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.202115190 (522h16m18.073198225s) [noise=true , address=145.79.227.7:4294    , piece_id=EURHN3JZWUE3J3UCPTUCSIFY62FGMZ555CLLQV3DGREDY57AGPYQ, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.211027030 (522h16m18.082110075s) [noise=true , address=145.79.227.25:4618   , piece_id=AIGHXCTF4VR44BQIV5SFNP5WNVB7UNZD2JGSDERGIIXZCJX57SEQ, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.223179019 (522h16m18.094262064s) [noise=true , address=145.79.227.47:4265   , piece_id=2C5A6NJTYAYDEGBJOYLMXMOOHRXR5HQEGZ4NLOOE5XJHY7IXRA6Q, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.409682317 (522h16m18.280765362s) [noise=true , address=77.90.31.21:4877     , piece_id=7EC3QLVJEE3XTGIYOZUYCZ5VOMEFEIMMGSDOORJSWXAMU42EXFQA, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.454868359 (522h16m18.325951404s) [noise=true , address=45.91.116.13:4156    , piece_id=NHOOPPW4CPTZSMU63KVDKAJSMQF5FAGNQZWJIN7GP5KRXHOWQ73Q, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.459736368 (522h16m18.330819414s) [noise=true , address=45.91.116.7:4011     , piece_id=VJEBW52OX5ZC3Y66N6JZYZVZHDX2ZDYYGR62YREUEWCX22QM74FQ, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.460076807 (522h16m18.331159852s) [noise=true , address=77.90.31.10:4520     , piece_id=GOBU6AFGRD2NSF3EK2NSVXH2PGFDH2XCC2YJCUUZNNHV25CN5QTQ, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.474866098 (522h16m18.345949144s) [noise=true , address=45.91.116.10:4083    , piece_id=VYZ5LCPNNDKZGQ4E3DKPLMI7Y4EACVZIWUSSBXMZ2K2NKL36MGUA, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.525058003 (522h16m18.396141058s) [noise=true , address=45.91.116.6:4973     , piece_id=E4RZUUW55HGZXQLNUSVCYIWFXAZML7AG6T5KEV6KTNI6BIA7NY4A, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.550081683 (522h16m18.421164738s) [noise=true , address=45.91.116.9:4262     , piece_id=GU2IXOE4O7PBYCXOBJDPPC5XLLTWEBPLFS26BGCY7OX2XXAI4QJQ, seg_pos={0 0}, upload=458165]: Done uploading piece. err: <nil>
Dec 27 09:45:18.550110417 (522h16m18.421193462s) [seg_pos={0 0}, upload=458165]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:45:18.550361626 (522h16m18.421444671s) [noise=true , address=77.90.31.12:4589     , piece_id=PXERZ4ZIS6LIQ6ONVH5PW57JRKSWF36AIYHGTEUOVEPJXMZNGHNA, seg_pos={0 0}, upload=458165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 27 09:45:18.550371364 (522h16m18.42145441s) [noise=true , address=77.90.31.12:4589     , piece_id=PXERZ4ZIS6LIQ6ONVH5PW57JRKSWF36AIYHGTEUOVEPJXMZNGHNA, seg_pos={0 0}, upload=458165]: Long tail cancellation due to optimal threshold
Dec 27 09:45:18.550495180 (522h16m18.421578215s) [noise=true , address=45.91.116.11:4119    , piece_id=UF75UBTJHR6I52T22SRS2IM72Z3CVIR7C2JFDBI57KEMVT7GBN5A, seg_pos={0 0}, upload=458165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:55522->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:55522->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55522->45.91.116.11:4119: use of closed network connection
Dec 27 09:45:18.550517242 (522h16m18.421600277s) [noise=true , address=45.91.116.11:4119    , piece_id=UF75UBTJHR6I52T22SRS2IM72Z3CVIR7C2JFDBI57KEMVT7GBN5A, seg_pos={0 0}, upload=458165]: Long tail cancellation due to optimal threshold
Dec 27 09:45:18.550672198 (522h16m18.421755243s) [noise=true , address=45.91.116.14:4191    , piece_id=RXCUMIEZGQRTILRMSVYH3QH3HF46XJV2RZXMJXSU7ZXZVEGCDRVA, seg_pos={0 0}, upload=458165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 09:45:18.550680533 (522h16m18.421763578s) [noise=true , address=45.91.116.14:4191    , piece_id=RXCUMIEZGQRTILRMSVYH3QH3HF46XJV2RZXMJXSU7ZXZVEGCDRVA, seg_pos={0 0}, upload=458165]: Long tail cancellation due to optimal threshold
Dec 27 09:45:18.550702695 (522h16m18.42178573s) [seg_pos={0 0}, upload=458165]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:45:18.550784601 (522h16m18.421867637s) [upload=458165]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:45:18.550789911 (522h16m18.421872957s) [upload=458165]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:45:18.832484490 (522h16m18.703567545s) [upload=458165]: Done uploading segments...
Dec 27 09:45:18.832510920 (522h16m18.703593955s) [upload=458165]: Upload finished. err: <nil>
Duration: 30.002999067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..................X........X.............X..................]
duration: 
40.829990083s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:45:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 40.830007206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...................X........X.............X.................]
duration: 
39.288289522s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:44:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 39.288319959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [.X........X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX..]
duration: 
5.884446921s
trace: 
https://jaeger.dev.storj.io/trace/34e5cdb8dc42a081
FAIL (2025-12-27 09:44:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA) context deadline exceeded
======LOGS======
Dec 27 09:42:31.605340773 (522h13m31.476423839s) [upload=458130]: Starting upload
Dec 27 09:42:31.605431135 (522h13m31.47651418s) [upload=458130]: Done starting upload
Dec 27 09:42:31.605549541 (522h13m31.476632576s) [upload=458130]: Uploading segments...
Dec 27 09:42:31.605739622 (522h13m31.476822658s) [upload=458130]: Got next segment. Inline: false
Dec 27 09:42:31.605773918 (522h13m31.476856953s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:42:31.605779108 (522h13m31.476862153s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:42:31.845531560 (522h13m31.716614595s) [seg_pos={0 0}, upload=458130]: Begin upload segment...
Dec 27 09:42:31.845577859 (522h13m31.716660894s) [seg_pos={0 0}, upload=458130]: Done begin upload segment.
Dec 27 09:42:31.845625499 (522h13m31.716708575s) [noise=true , address=145.79.227.39:4013   , piece_id=RMBX64WJAFKDDCRYWFTBHPIQXC6EQMRZBLQDXEJDSAUIJ2D3QAVQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.845641390 (522h13m31.716724435s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.845844456 (522h13m31.716927491s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846274285 (522h13m31.71735732s) [noise=true , address=45.91.116.5:4300     , piece_id=EI7IR4QTJZUO3QDSFCOG4FOHU2KCB7GYXJBIEHAJHAJEG5WNLLFA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846628209 (522h13m31.717711244s) [noise=true , address=77.90.31.11:4553     , piece_id=DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846946986 (522h13m31.718030021s) [noise=true , address=45.91.116.9:4264     , piece_id=K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.847248821 (522h13m31.718331856s) [noise=true , address=45.91.116.10:4082    , piece_id=NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.847663040 (522h13m31.718746075s) [noise=true , address=145.79.227.7:4294    , piece_id=Q7JSJYT5BT4LK3E3MNNHS7W2N65WO3OO25TV46YY6PUH6AGQZCBA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.849149600 (522h13m31.720232635s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.850608688 (522h13m31.721691733s) [noise=true , address=77.90.31.15:4697     , piece_id=ZU2FOZPB35YOYAX7FMQFIVFAZFRO4IRNXQVFKWHJ7NKYM3D23WDQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.851867565 (522h13m31.72295061s) [noise=true , address=145.79.227.17:4972   , piece_id=OBWG77V2N5TO4JMW4MFOEPB2AGEFGDEQ35F6NYQ5TY3LIP6BRVDQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.853105653 (522h13m31.724188688s) [noise=true , address=145.79.227.51:4409   , piece_id=MMIEHCQZTKZHTS476YG4CNM5KGE5OWBCFWPQH2CP64KLLYANLNXQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.854850375 (522h13m31.72593342s) [noise=true , address=45.91.116.12:4228    , piece_id=PQW2RGX4OI42NSXY4XARY2PZ3QR46YTPLEFS2OED4YXF75YSTWDA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.861414565 (522h13m31.7324976s) [noise=true , address=145.79.227.25:4618   , piece_id=ELOPBMVXVC27XAJ6L5LJXY3RVFKNWPZVWSG2GHFDGFIVSX7EA2DQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.861552949 (522h13m31.732636024s) [noise=true , address=145.79.227.12:4765   , piece_id=EVTBQKWOAP5X7G5BHXNKK5I3ICNFVR7WDMV2B3IIB3V6UEYHRAZQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.864917344 (522h13m31.736000389s) [noise=true , address=145.79.227.52:4445   , piece_id=MOTO32OVE3VTXP4S4TJKQLS6PF3XZI6ZNHNPQUZFYS3TNTB2H66Q, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:33.571475005 (522h13m33.442558041s) [upload=458130]: Next returned nil segment
Dec 27 09:42:33.571488100 (522h13m33.442571145s) [upload=458130]: Waiting for error group managing segments...
Dec 27 09:42:33.847206018 (522h13m33.718289053s) [noise=true , address=145.79.227.39:4013   , piece_id=RMBX64WJAFKDDCRYWFTBHPIQXC6EQMRZBLQDXEJDSAUIJ2D3QAVQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.855139746 (522h13m33.726222791s) [noise=true , address=145.79.227.7:4294    , piece_id=Q7JSJYT5BT4LK3E3MNNHS7W2N65WO3OO25TV46YY6PUH6AGQZCBA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.862340357 (522h13m33.733423402s) [noise=true , address=145.79.227.17:4972   , piece_id=OBWG77V2N5TO4JMW4MFOEPB2AGEFGDEQ35F6NYQ5TY3LIP6BRVDQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.871913185 (522h13m33.74299622s) [noise=true , address=145.79.227.51:4409   , piece_id=MMIEHCQZTKZHTS476YG4CNM5KGE5OWBCFWPQH2CP64KLLYANLNXQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.874906104 (522h13m33.745989149s) [noise=true , address=145.79.227.52:4445   , piece_id=MOTO32OVE3VTXP4S4TJKQLS6PF3XZI6ZNHNPQUZFYS3TNTB2H66Q, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.883080821 (522h13m33.754163866s) [noise=true , address=145.79.227.12:4765   , piece_id=EVTBQKWOAP5X7G5BHXNKK5I3ICNFVR7WDMV2B3IIB3V6UEYHRAZQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.886995464 (522h13m33.758078499s) [noise=true , address=145.79.227.25:4618   , piece_id=ELOPBMVXVC27XAJ6L5LJXY3RVFKNWPZVWSG2GHFDGFIVSX7EA2DQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.041671366 (522h13m33.912754401s) [noise=true , address=77.90.31.15:4697     , piece_id=ZU2FOZPB35YOYAX7FMQFIVFAZFRO4IRNXQVFKWHJ7NKYM3D23WDQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.116533407 (522h13m33.987616452s) [noise=true , address=77.90.31.11:4553     , piece_id=DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.117634885 (522h13m33.98871792s) [noise=true , address=45.91.116.9:4264     , piece_id=K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.122368128 (522h13m33.993451163s) [noise=true , address=45.91.116.12:4228    , piece_id=PQW2RGX4OI42NSXY4XARY2PZ3QR46YTPLEFS2OED4YXF75YSTWDA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.170857821 (522h13m34.041940856s) [noise=true , address=45.91.116.5:4300     , piece_id=EI7IR4QTJZUO3QDSFCOG4FOHU2KCB7GYXJBIEHAJHAJEG5WNLLFA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.198671593 (522h13m34.069754628s) [noise=true , address=45.91.116.10:4082    , piece_id=NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.198690800 (522h13m34.069773835s) [seg_pos={0 0}, upload=458130]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:42:34.198852978 (522h13m34.069936013s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 27 09:42:34.198858510 (522h13m34.069941555s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.198974370 (522h13m34.070057415s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 27 09:42:34.198983086 (522h13m34.070066121s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.199057908 (522h13m34.070140954s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection
Dec 27 09:42:34.199064081 (522h13m34.070147126s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.199079570 (522h13m34.070162605s) [seg_pos={0 0}, upload=458130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:42:34.199149634 (522h13m34.070232679s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:42:34.199154302 (522h13m34.070237337s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:42:34.506504484 (522h13m34.377587519s) [upload=458130]: Done uploading segments...
Dec 27 09:42:34.506526987 (522h13m34.377610022s) [upload=458130]: Upload finished. err: <nil>
Duration: 5.88448222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [....................X........X.............X................]
duration: 
32.71020181s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:43:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 32.710240903s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [..X........X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX.]
duration: 
5.941898052s
trace: 
https://jaeger.dev.storj.io/trace/34e5cdb8dc42a081
FAIL (2025-12-27 09:43:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA) context deadline exceeded
======LOGS======
Dec 27 09:42:31.605340773 (522h13m31.476423839s) [upload=458130]: Starting upload
Dec 27 09:42:31.605431135 (522h13m31.47651418s) [upload=458130]: Done starting upload
Dec 27 09:42:31.605549541 (522h13m31.476632576s) [upload=458130]: Uploading segments...
Dec 27 09:42:31.605739622 (522h13m31.476822658s) [upload=458130]: Got next segment. Inline: false
Dec 27 09:42:31.605773918 (522h13m31.476856953s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:42:31.605779108 (522h13m31.476862153s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:42:31.845531560 (522h13m31.716614595s) [seg_pos={0 0}, upload=458130]: Begin upload segment...
Dec 27 09:42:31.845577859 (522h13m31.716660894s) [seg_pos={0 0}, upload=458130]: Done begin upload segment.
Dec 27 09:42:31.845625499 (522h13m31.716708575s) [noise=true , address=145.79.227.39:4013   , piece_id=RMBX64WJAFKDDCRYWFTBHPIQXC6EQMRZBLQDXEJDSAUIJ2D3QAVQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.845641390 (522h13m31.716724435s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.845844456 (522h13m31.716927491s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846274285 (522h13m31.71735732s) [noise=true , address=45.91.116.5:4300     , piece_id=EI7IR4QTJZUO3QDSFCOG4FOHU2KCB7GYXJBIEHAJHAJEG5WNLLFA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846628209 (522h13m31.717711244s) [noise=true , address=77.90.31.11:4553     , piece_id=DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846946986 (522h13m31.718030021s) [noise=true , address=45.91.116.9:4264     , piece_id=K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.847248821 (522h13m31.718331856s) [noise=true , address=45.91.116.10:4082    , piece_id=NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.847663040 (522h13m31.718746075s) [noise=true , address=145.79.227.7:4294    , piece_id=Q7JSJYT5BT4LK3E3MNNHS7W2N65WO3OO25TV46YY6PUH6AGQZCBA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.849149600 (522h13m31.720232635s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.850608688 (522h13m31.721691733s) [noise=true , address=77.90.31.15:4697     , piece_id=ZU2FOZPB35YOYAX7FMQFIVFAZFRO4IRNXQVFKWHJ7NKYM3D23WDQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.851867565 (522h13m31.72295061s) [noise=true , address=145.79.227.17:4972   , piece_id=OBWG77V2N5TO4JMW4MFOEPB2AGEFGDEQ35F6NYQ5TY3LIP6BRVDQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.853105653 (522h13m31.724188688s) [noise=true , address=145.79.227.51:4409   , piece_id=MMIEHCQZTKZHTS476YG4CNM5KGE5OWBCFWPQH2CP64KLLYANLNXQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.854850375 (522h13m31.72593342s) [noise=true , address=45.91.116.12:4228    , piece_id=PQW2RGX4OI42NSXY4XARY2PZ3QR46YTPLEFS2OED4YXF75YSTWDA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.861414565 (522h13m31.7324976s) [noise=true , address=145.79.227.25:4618   , piece_id=ELOPBMVXVC27XAJ6L5LJXY3RVFKNWPZVWSG2GHFDGFIVSX7EA2DQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.861552949 (522h13m31.732636024s) [noise=true , address=145.79.227.12:4765   , piece_id=EVTBQKWOAP5X7G5BHXNKK5I3ICNFVR7WDMV2B3IIB3V6UEYHRAZQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.864917344 (522h13m31.736000389s) [noise=true , address=145.79.227.52:4445   , piece_id=MOTO32OVE3VTXP4S4TJKQLS6PF3XZI6ZNHNPQUZFYS3TNTB2H66Q, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:33.571475005 (522h13m33.442558041s) [upload=458130]: Next returned nil segment
Dec 27 09:42:33.571488100 (522h13m33.442571145s) [upload=458130]: Waiting for error group managing segments...
Dec 27 09:42:33.847206018 (522h13m33.718289053s) [noise=true , address=145.79.227.39:4013   , piece_id=RMBX64WJAFKDDCRYWFTBHPIQXC6EQMRZBLQDXEJDSAUIJ2D3QAVQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.855139746 (522h13m33.726222791s) [noise=true , address=145.79.227.7:4294    , piece_id=Q7JSJYT5BT4LK3E3MNNHS7W2N65WO3OO25TV46YY6PUH6AGQZCBA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.862340357 (522h13m33.733423402s) [noise=true , address=145.79.227.17:4972   , piece_id=OBWG77V2N5TO4JMW4MFOEPB2AGEFGDEQ35F6NYQ5TY3LIP6BRVDQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.871913185 (522h13m33.74299622s) [noise=true , address=145.79.227.51:4409   , piece_id=MMIEHCQZTKZHTS476YG4CNM5KGE5OWBCFWPQH2CP64KLLYANLNXQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.874906104 (522h13m33.745989149s) [noise=true , address=145.79.227.52:4445   , piece_id=MOTO32OVE3VTXP4S4TJKQLS6PF3XZI6ZNHNPQUZFYS3TNTB2H66Q, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.883080821 (522h13m33.754163866s) [noise=true , address=145.79.227.12:4765   , piece_id=EVTBQKWOAP5X7G5BHXNKK5I3ICNFVR7WDMV2B3IIB3V6UEYHRAZQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.886995464 (522h13m33.758078499s) [noise=true , address=145.79.227.25:4618   , piece_id=ELOPBMVXVC27XAJ6L5LJXY3RVFKNWPZVWSG2GHFDGFIVSX7EA2DQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.041671366 (522h13m33.912754401s) [noise=true , address=77.90.31.15:4697     , piece_id=ZU2FOZPB35YOYAX7FMQFIVFAZFRO4IRNXQVFKWHJ7NKYM3D23WDQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.116533407 (522h13m33.987616452s) [noise=true , address=77.90.31.11:4553     , piece_id=DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.117634885 (522h13m33.98871792s) [noise=true , address=45.91.116.9:4264     , piece_id=K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.122368128 (522h13m33.993451163s) [noise=true , address=45.91.116.12:4228    , piece_id=PQW2RGX4OI42NSXY4XARY2PZ3QR46YTPLEFS2OED4YXF75YSTWDA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.170857821 (522h13m34.041940856s) [noise=true , address=45.91.116.5:4300     , piece_id=EI7IR4QTJZUO3QDSFCOG4FOHU2KCB7GYXJBIEHAJHAJEG5WNLLFA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.198671593 (522h13m34.069754628s) [noise=true , address=45.91.116.10:4082    , piece_id=NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.198690800 (522h13m34.069773835s) [seg_pos={0 0}, upload=458130]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:42:34.198852978 (522h13m34.069936013s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 27 09:42:34.198858510 (522h13m34.069941555s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.198974370 (522h13m34.070057415s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 27 09:42:34.198983086 (522h13m34.070066121s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.199057908 (522h13m34.070140954s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection
Dec 27 09:42:34.199064081 (522h13m34.070147126s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.199079570 (522h13m34.070162605s) [seg_pos={0 0}, upload=458130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:42:34.199149634 (522h13m34.070232679s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:42:34.199154302 (522h13m34.070237337s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:42:34.506504484 (522h13m34.377587519s) [upload=458130]: Done uploading segments...
Dec 27 09:42:34.506526987 (522h13m34.377610022s) [upload=458130]: Upload finished. err: <nil>
Duration: 5.941924201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [...X........X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....XX]
duration: 
30.002627266s
trace: 
https://jaeger.dev.storj.io/trace/34e5cdb8dc42a081
FAIL (2025-12-27 09:43:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA) context deadline exceeded
======LOGS======
Dec 27 09:42:31.605340773 (522h13m31.476423839s) [upload=458130]: Starting upload
Dec 27 09:42:31.605431135 (522h13m31.47651418s) [upload=458130]: Done starting upload
Dec 27 09:42:31.605549541 (522h13m31.476632576s) [upload=458130]: Uploading segments...
Dec 27 09:42:31.605739622 (522h13m31.476822658s) [upload=458130]: Got next segment. Inline: false
Dec 27 09:42:31.605773918 (522h13m31.476856953s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:42:31.605779108 (522h13m31.476862153s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:42:31.845531560 (522h13m31.716614595s) [seg_pos={0 0}, upload=458130]: Begin upload segment...
Dec 27 09:42:31.845577859 (522h13m31.716660894s) [seg_pos={0 0}, upload=458130]: Done begin upload segment.
Dec 27 09:42:31.845625499 (522h13m31.716708575s) [noise=true , address=145.79.227.39:4013   , piece_id=RMBX64WJAFKDDCRYWFTBHPIQXC6EQMRZBLQDXEJDSAUIJ2D3QAVQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.845641390 (522h13m31.716724435s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.845844456 (522h13m31.716927491s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846274285 (522h13m31.71735732s) [noise=true , address=45.91.116.5:4300     , piece_id=EI7IR4QTJZUO3QDSFCOG4FOHU2KCB7GYXJBIEHAJHAJEG5WNLLFA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846628209 (522h13m31.717711244s) [noise=true , address=77.90.31.11:4553     , piece_id=DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.846946986 (522h13m31.718030021s) [noise=true , address=45.91.116.9:4264     , piece_id=K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.847248821 (522h13m31.718331856s) [noise=true , address=45.91.116.10:4082    , piece_id=NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.847663040 (522h13m31.718746075s) [noise=true , address=145.79.227.7:4294    , piece_id=Q7JSJYT5BT4LK3E3MNNHS7W2N65WO3OO25TV46YY6PUH6AGQZCBA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.849149600 (522h13m31.720232635s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.850608688 (522h13m31.721691733s) [noise=true , address=77.90.31.15:4697     , piece_id=ZU2FOZPB35YOYAX7FMQFIVFAZFRO4IRNXQVFKWHJ7NKYM3D23WDQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.851867565 (522h13m31.72295061s) [noise=true , address=145.79.227.17:4972   , piece_id=OBWG77V2N5TO4JMW4MFOEPB2AGEFGDEQ35F6NYQ5TY3LIP6BRVDQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.853105653 (522h13m31.724188688s) [noise=true , address=145.79.227.51:4409   , piece_id=MMIEHCQZTKZHTS476YG4CNM5KGE5OWBCFWPQH2CP64KLLYANLNXQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.854850375 (522h13m31.72593342s) [noise=true , address=45.91.116.12:4228    , piece_id=PQW2RGX4OI42NSXY4XARY2PZ3QR46YTPLEFS2OED4YXF75YSTWDA, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.861414565 (522h13m31.7324976s) [noise=true , address=145.79.227.25:4618   , piece_id=ELOPBMVXVC27XAJ6L5LJXY3RVFKNWPZVWSG2GHFDGFIVSX7EA2DQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.861552949 (522h13m31.732636024s) [noise=true , address=145.79.227.12:4765   , piece_id=EVTBQKWOAP5X7G5BHXNKK5I3ICNFVR7WDMV2B3IIB3V6UEYHRAZQ, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:31.864917344 (522h13m31.736000389s) [noise=true , address=145.79.227.52:4445   , piece_id=MOTO32OVE3VTXP4S4TJKQLS6PF3XZI6ZNHNPQUZFYS3TNTB2H66Q, seg_pos={0 0}, upload=458130]: Uploading piece...
Dec 27 09:42:33.571475005 (522h13m33.442558041s) [upload=458130]: Next returned nil segment
Dec 27 09:42:33.571488100 (522h13m33.442571145s) [upload=458130]: Waiting for error group managing segments...
Dec 27 09:42:33.847206018 (522h13m33.718289053s) [noise=true , address=145.79.227.39:4013   , piece_id=RMBX64WJAFKDDCRYWFTBHPIQXC6EQMRZBLQDXEJDSAUIJ2D3QAVQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.855139746 (522h13m33.726222791s) [noise=true , address=145.79.227.7:4294    , piece_id=Q7JSJYT5BT4LK3E3MNNHS7W2N65WO3OO25TV46YY6PUH6AGQZCBA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.862340357 (522h13m33.733423402s) [noise=true , address=145.79.227.17:4972   , piece_id=OBWG77V2N5TO4JMW4MFOEPB2AGEFGDEQ35F6NYQ5TY3LIP6BRVDQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.871913185 (522h13m33.74299622s) [noise=true , address=145.79.227.51:4409   , piece_id=MMIEHCQZTKZHTS476YG4CNM5KGE5OWBCFWPQH2CP64KLLYANLNXQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.874906104 (522h13m33.745989149s) [noise=true , address=145.79.227.52:4445   , piece_id=MOTO32OVE3VTXP4S4TJKQLS6PF3XZI6ZNHNPQUZFYS3TNTB2H66Q, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.883080821 (522h13m33.754163866s) [noise=true , address=145.79.227.12:4765   , piece_id=EVTBQKWOAP5X7G5BHXNKK5I3ICNFVR7WDMV2B3IIB3V6UEYHRAZQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:33.886995464 (522h13m33.758078499s) [noise=true , address=145.79.227.25:4618   , piece_id=ELOPBMVXVC27XAJ6L5LJXY3RVFKNWPZVWSG2GHFDGFIVSX7EA2DQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.041671366 (522h13m33.912754401s) [noise=true , address=77.90.31.15:4697     , piece_id=ZU2FOZPB35YOYAX7FMQFIVFAZFRO4IRNXQVFKWHJ7NKYM3D23WDQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.116533407 (522h13m33.987616452s) [noise=true , address=77.90.31.11:4553     , piece_id=DEFRMLZHWYMB2IWBPN67ZR4ITEXO7LSQZSHOS6S5UKIARCCPOS6A, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.117634885 (522h13m33.98871792s) [noise=true , address=45.91.116.9:4264     , piece_id=K2RRLWMPXVWFTJEB3WUPU2WHM4X3NEMURX7N7ZNHDG324366CBHQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.122368128 (522h13m33.993451163s) [noise=true , address=45.91.116.12:4228    , piece_id=PQW2RGX4OI42NSXY4XARY2PZ3QR46YTPLEFS2OED4YXF75YSTWDA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.170857821 (522h13m34.041940856s) [noise=true , address=45.91.116.5:4300     , piece_id=EI7IR4QTJZUO3QDSFCOG4FOHU2KCB7GYXJBIEHAJHAJEG5WNLLFA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.198671593 (522h13m34.069754628s) [noise=true , address=45.91.116.10:4082    , piece_id=NZYIK6CHKJBEC7EQNDDUV65WHUBRZOLJWAAIZBW5UD5E2NV4CCWA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: <nil>
Dec 27 09:42:34.198690800 (522h13m34.069773835s) [seg_pos={0 0}, upload=458130]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:42:34.198852978 (522h13m34.069936013s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 27 09:42:34.198858510 (522h13m34.069941555s) [noise=true , address=145.79.227.44:4195   , piece_id=T3ZM6UT6EIMFH2NQOS5SPL5JTXQEXNW3YDC7C2SFS3ESUB7NEUBA, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.198974370 (522h13m34.070057415s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 27 09:42:34.198983086 (522h13m34.070066121s) [noise=true , address=45.91.116.6:4973     , piece_id=XIDYFJG4VTGWW6IR3MHLQ2XUPRZDIWPCAL6LVTJQR2H7SCZQ7JLQ, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.199057908 (522h13m34.070140954s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60496->77.90.31.21:4880: use of closed network connection
Dec 27 09:42:34.199064081 (522h13m34.070147126s) [noise=true , address=77.90.31.21:4880     , piece_id=DEGYH3UAJOQZO5PZUVSFO4HWIXKDGTMVAXG6UCIBSMFDDQUM5BNQ, seg_pos={0 0}, upload=458130]: Long tail cancellation due to optimal threshold
Dec 27 09:42:34.199079570 (522h13m34.070162605s) [seg_pos={0 0}, upload=458130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:42:34.199149634 (522h13m34.070232679s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:42:34.199154302 (522h13m34.070237337s) [upload=458130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:42:34.506504484 (522h13m34.377587519s) [upload=458130]: Done uploading segments...
Dec 27 09:42:34.506526987 (522h13m34.377610022s) [upload=458130]: Upload finished. err: <nil>
Duration: 30.002656631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.....................X........X.............X...............]
duration: 
46.989751398s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:42:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 46.989771165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [X...X........X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....X]
duration: 
30.003639414s
trace: 
https://jaeger.dev.storj.io/trace/648f43982f24e4fc
FAIL (2025-12-27 09:42:20 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 4CIN7PJIHPC5CJGKCTE4PEOI3Q5CGHTNUWQJAEHIALXWSKB4LKPA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7GVBR3CWZW7I4B6P3HUCV42TPF42LT6MINIXCWEMMJK37JVALPFA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: SUESUWOQF6UMUK7POEP6MDE753ZUPFLIS4BRGHPLDN25FX5W3ASA) context deadline exceeded
======LOGS======
937113 (522h12m51.394020158s) [upload=458120]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:41:51.763052677 (522h12m51.634135722s) [seg_pos={0 0}, upload=458120]: Begin upload segment...
Dec 27 09:41:51.763110065 (522h12m51.63419311s) [seg_pos={0 0}, upload=458120]: Done begin upload segment.
Dec 27 09:41:51.763141455 (522h12m51.63422449s) [noise=true , address=77.90.31.10:4518     , piece_id=4CIN7PJIHPC5CJGKCTE4PEOI3Q5CGHTNUWQJAEHIALXWSKB4LKPA, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.763147537 (522h12m51.634230582s) [noise=true , address=45.91.116.12:4228    , piece_id=T374EX24SVZR44OB44S36RFIBIOT35FFMJFQVQKWMH576NFJPAGQ, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.763627391 (522h12m51.634710436s) [noise=true , address=145.79.227.7:4294    , piece_id=OLPDLLANHKHTULAUVM76DY6IHGBGYZKK2ALVKBYFZCUUESHYRIHQ, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.764022754 (522h12m51.635105799s) [noise=true , address=145.79.227.45:4229   , piece_id=KUNRMOJ2NYXND42QJNCE2LHHHGYCUTSWEU3Q4BSH4J37KAWCIUEA, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.764410502 (522h12m51.635493537s) [noise=true , address=145.79.227.49:4339   , piece_id=UJT4D7Z6WG5S5NXFHVXUWSZBAAYGXARTXHIETGTAJYBC3UC5CNWQ, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.764804743 (522h12m51.635887788s) [noise=true , address=145.79.227.41:4087   , piece_id=FTQUWXE7CMANPS5SJXYEVQCE7R3465G5I3AREFOFJVDBK5N3J7ZQ, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.765225784 (522h12m51.636308829s) [noise=true , address=45.91.116.5:4299     , piece_id=7GVBR3CWZW7I4B6P3HUCV42TPF42LT6MINIXCWEMMJK37JVALPFA, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.765731919 (522h12m51.636814974s) [noise=true , address=45.91.116.7:4011     , piece_id=734KZHTPROCNSV4NW7TQKHKKJ2B5OOEH2KN2DWGQTRXCA47YEA6Q, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.766923718 (522h12m51.638006753s) [noise=true , address=45.91.116.11:4120    , piece_id=PFHDFI4NM5QME4Z4MRXSKO5JGE4GXR4ZCYVYUR7YRR5ADG3YBMXA, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.768340766 (522h12m51.639423801s) [noise=true , address=77.90.31.13:4629     , piece_id=DUC3ARVIKCTXYDX7OXWFHAKLIF4LRDYKBQTBIDSLQXKFBQLZD6SQ, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.769760409 (522h12m51.640843454s) [noise=true , address=45.91.116.14:4189    , piece_id=SUESUWOQF6UMUK7POEP6MDE753ZUPFLIS4BRGHPLDN25FX5W3ASA, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.771451830 (522h12m51.642534875s) [noise=true , address=77.90.31.15:4700     , piece_id=QERXC5RDTHKHEEVZEBKDBEP5WEK7Y7CNX6BAJXIEFVIONZXTRBBQ, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.773053530 (522h12m51.644136565s) [noise=true , address=145.79.227.17:4970   , piece_id=GXRFZ52LBO5GCVNNUX6HT2HYXYTDZXBIER5URHLLWS7EZDYDWNFA, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.774589064 (522h12m51.645672109s) [noise=true , address=145.79.227.50:4375   , piece_id=VFMBM52N6KPL4GNGWYEH4WBFC2YXM3GH2LQYUIROIF6LWCHFP5DQ, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.775948823 (522h12m51.647031858s) [noise=true , address=45.91.116.4:4939     , piece_id=SW36RPTWFHIOBFP4RQ5YLRL4GXQPNMMRNM3B4SIA5MUVDSUUDXYA, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:51.777114854 (522h12m51.648197889s) [noise=true , address=145.79.227.46:4601   , piece_id=TN5KKERGEF57P4BKQ3WLRZBQXRJSCLVMDZAJFNCSHXB3KVQLMNQQ, seg_pos={0 0}, upload=458120]: Uploading piece...
Dec 27 09:41:53.478921216 (522h12m53.350004251s) [upload=458120]: Next returned nil segment
Dec 27 09:41:53.478953999 (522h12m53.350037044s) [upload=458120]: Waiting for error group managing segments...
Dec 27 09:41:53.774337634 (522h12m53.645420679s) [noise=true , address=145.79.227.41:4087   , piece_id=FTQUWXE7CMANPS5SJXYEVQCE7R3465G5I3AREFOFJVDBK5N3J7ZQ, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.775328711 (522h12m53.646411757s) [noise=true , address=145.79.227.45:4229   , piece_id=KUNRMOJ2NYXND42QJNCE2LHHHGYCUTSWEU3Q4BSH4J37KAWCIUEA, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.777207900 (522h12m53.648290935s) [noise=true , address=145.79.227.49:4339   , piece_id=UJT4D7Z6WG5S5NXFHVXUWSZBAAYGXARTXHIETGTAJYBC3UC5CNWQ, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.779203289 (522h12m53.650286334s) [noise=true , address=145.79.227.46:4601   , piece_id=TN5KKERGEF57P4BKQ3WLRZBQXRJSCLVMDZAJFNCSHXB3KVQLMNQQ, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.789900578 (522h12m53.660983613s) [noise=true , address=145.79.227.7:4294    , piece_id=OLPDLLANHKHTULAUVM76DY6IHGBGYZKK2ALVKBYFZCUUESHYRIHQ, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.796453246 (522h12m53.667536281s) [noise=true , address=145.79.227.17:4970   , piece_id=GXRFZ52LBO5GCVNNUX6HT2HYXYTDZXBIER5URHLLWS7EZDYDWNFA, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.809188938 (522h12m53.680271973s) [noise=true , address=145.79.227.50:4375   , piece_id=VFMBM52N6KPL4GNGWYEH4WBFC2YXM3GH2LQYUIROIF6LWCHFP5DQ, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.963958718 (522h12m53.835041763s) [noise=true , address=77.90.31.13:4629     , piece_id=DUC3ARVIKCTXYDX7OXWFHAKLIF4LRDYKBQTBIDSLQXKFBQLZD6SQ, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.990493134 (522h12m53.861576169s) [noise=true , address=45.91.116.7:4011     , piece_id=734KZHTPROCNSV4NW7TQKHKKJ2B5OOEH2KN2DWGQTRXCA47YEA6Q, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:53.993022480 (522h12m53.864105515s) [noise=true , address=77.90.31.10:4518     , piece_id=4CIN7PJIHPC5CJGKCTE4PEOI3Q5CGHTNUWQJAEHIALXWSKB4LKPA, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:54.046269673 (522h12m53.917352738s) [noise=true , address=45.91.116.4:4939     , piece_id=SW36RPTWFHIOBFP4RQ5YLRL4GXQPNMMRNM3B4SIA5MUVDSUUDXYA, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:54.049578784 (522h12m53.920661819s) [noise=true , address=45.91.116.5:4299     , piece_id=7GVBR3CWZW7I4B6P3HUCV42TPF42LT6MINIXCWEMMJK37JVALPFA, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:54.189142429 (522h12m54.060225464s) [noise=true , address=45.91.116.14:4189    , piece_id=SUESUWOQF6UMUK7POEP6MDE753ZUPFLIS4BRGHPLDN25FX5W3ASA, seg_pos={0 0}, upload=458120]: Done uploading piece. err: <nil>
Dec 27 09:41:54.189162427 (522h12m54.060245462s) [seg_pos={0 0}, upload=458120]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:41:54.189334375 (522h12m54.06041741s) [noise=true , address=45.91.116.11:4120    , piece_id=PFHDFI4NM5QME4Z4MRXSKO5JGE4GXR4ZCYVYUR7YRR5ADG3YBMXA, seg_pos={0 0}, upload=458120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60542->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60542->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60542->45.91.116.11:4120: use of closed network connection
Dec 27 09:41:54.189339995 (522h12m54.06042303s) [noise=true , address=45.91.116.11:4120    , piece_id=PFHDFI4NM5QME4Z4MRXSKO5JGE4GXR4ZCYVYUR7YRR5ADG3YBMXA, seg_pos={0 0}, upload=458120]: Long tail cancellation due to optimal threshold
Dec 27 09:41:54.189513045 (522h12m54.06059609s) [noise=true , address=77.90.31.15:4700     , piece_id=QERXC5RDTHKHEEVZEBKDBEP5WEK7Y7CNX6BAJXIEFVIONZXTRBBQ, seg_pos={0 0}, upload=458120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:36386->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:36386->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36386->77.90.31.15:4700: use of closed network connection
Dec 27 09:41:54.189528664 (522h12m54.060611699s) [noise=true , address=77.90.31.15:4700     , piece_id=QERXC5RDTHKHEEVZEBKDBEP5WEK7Y7CNX6BAJXIEFVIONZXTRBBQ, seg_pos={0 0}, upload=458120]: Long tail cancellation due to optimal threshold
Dec 27 09:41:54.189643584 (522h12m54.060726619s) [noise=true , address=45.91.116.12:4228    , piece_id=T374EX24SVZR44OB44S36RFIBIOT35FFMJFQVQKWMH576NFJPAGQ, seg_pos={0 0}, upload=458120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:55720->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:55720->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->45.91.116.12:4228: use of closed network connection
Dec 27 09:41:54.189651508 (522h12m54.060734544s) [noise=true , address=45.91.116.12:4228    , piece_id=T374EX24SVZR44OB44S36RFIBIOT35FFMJFQVQKWMH576NFJPAGQ, seg_pos={0 0}, upload=458120]: Long tail cancellation due to optimal threshold
Dec 27 09:41:54.189667599 (522h12m54.060750644s) [seg_pos={0 0}, upload=458120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:41:54.189743653 (522h12m54.060826688s) [upload=458120]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:41:54.189748423 (522h12m54.060831458s) [upload=458120]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:41:54.472675468 (522h12m54.343758503s) [upload=458120]: Done uploading segments...
Dec 27 09:41:54.472692821 (522h12m54.343775856s) [upload=458120]: Upload finished. err: <nil>
Duration: 30.003659152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [......................X........X.............X..............]
duration: 
59.62540339s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:42:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 59.625435391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.X...X........X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX....]
duration: 
6.244487427s
trace: 
https://jaeger.dev.storj.io/trace/1906a6556cd44aef
FAIL (2025-12-27 09:41:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: JT4P2BNQFGWVBQGLXSS75KAP32CMDH6GAMJBWYHTBSPSXRFWABAQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: DX6TFOHKFRW37ES7ZSLV2QIXGSWDJ6TNRFRMG6MAH4LMSOO6PCNQ) context deadline exceeded
======LOGS======
Dec 27 09:38:29.005502783 (522h9m28.876585838s) [upload=458069]: Starting upload
Dec 27 09:38:29.005557136 (522h9m28.876640191s) [upload=458069]: Done starting upload
Dec 27 09:38:29.005690470 (522h9m28.876773515s) [upload=458069]: Uploading segments...
Dec 27 09:38:29.005855304 (522h9m28.876938359s) [upload=458069]: Got next segment. Inline: false
Dec 27 09:38:29.005893106 (522h9m28.876976151s) [upload=458069]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:38:29.005898687 (522h9m28.876981722s) [upload=458069]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:38:29.252809692 (522h9m29.123892727s) [seg_pos={0 0}, upload=458069]: Begin upload segment...
Dec 27 09:38:29.252869115 (522h9m29.12395215s) [seg_pos={0 0}, upload=458069]: Done begin upload segment.
Dec 27 09:38:29.252910173 (522h9m29.123993218s) [noise=true , address=145.79.227.49:4339   , piece_id=WIMRJNYLPIKBXUL7UWAFVAFHWKDX524VIX72QB4HTNWCCCTTFM5Q, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.252957102 (522h9m29.124040147s) [noise=true , address=77.90.31.15:4697     , piece_id=DX6TFOHKFRW37ES7ZSLV2QIXGSWDJ6TNRFRMG6MAH4LMSOO6PCNQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.253628521 (522h9m29.124711566s) [noise=true , address=45.91.116.10:4081    , piece_id=CR7MTAY6HK3K3ET7QOUYJGSS766Z4OWWUX4PZRHIA3TDAEYH3BIA, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.254253771 (522h9m29.125336816s) [noise=true , address=77.90.31.18:4805     , piece_id=KCFRWQJTVJ5TCUTKYTZOSRZ3HJ3HHZJMN6Q4XWOLAFQHIQQU5HEQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.254723125 (522h9m29.125806171s) [noise=true , address=145.79.227.37:4567   , piece_id=ZKQ7F32A6F23MBBLR6TATQSIE2YN5E2GVPDLXNCJKJ7LEB27TVUA, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.255160728 (522h9m29.126243763s) [noise=true , address=45.91.116.4:4937     , piece_id=O26FQGZXDLFTXAE3VN4XNPIGUXU5HKIHP4DX3BLQMB7ACEZMY5RQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.255607519 (522h9m29.126690564s) [noise=true , address=145.79.227.29:4943   , piece_id=OCZB6RSLV3VRZJCNH2XTS7QRPB7Y7GWIAVWSW7LL6P2H57SVFTUQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.256099706 (522h9m29.127182751s) [noise=true , address=145.79.227.28:4727   , piece_id=KCODK42JTX4GNR2KZKIHFQ2HD4TTNTBAE2UOIHMC4SCCQL4O2AIQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.257350708 (522h9m29.128433743s) [noise=true , address=145.79.227.30:4762   , piece_id=VYNQR6MIQIO7IOIOG7PZ2QDHJZZ3PA67AANQXMFBTUUSVOXZZAHQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.257748887 (522h9m29.128831922s) [noise=true , address=145.79.227.25:4618   , piece_id=MW2DI5XCWKAK24AUWVVLVPCJXQGG3C6BV47DIUUAZVVJYSVJUAHA, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.258982876 (522h9m29.130065911s) [noise=true , address=145.79.227.39:4015   , piece_id=Z5W6PU6YLBZDFZT6EAWEPP6ZMDLZISBOXRXR6AO7FZCXHBMAOXLA, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.260452795 (522h9m29.13153583s) [noise=true , address=45.91.116.13:4153    , piece_id=JT4P2BNQFGWVBQGLXSS75KAP32CMDH6GAMJBWYHTBSPSXRFWABAQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.262086436 (522h9m29.133169481s) [noise=true , address=145.79.227.46:4601   , piece_id=AAN4BKWIUOOI6F4KJJDYVZ33UFB7HTUNY3L2YWNT5APDPJLZNY4Q, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.263541056 (522h9m29.134624091s) [noise=true , address=145.79.227.7:4295    , piece_id=A2YMVUHMSQQNFANVSATKDOJXY2ZOI7KTCCMVVTCLNW6C2Y7CP6WQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.264445949 (522h9m29.135528994s) [noise=true , address=145.79.227.3:4150    , piece_id=P5KXLW74W4FQKI3D5YLBBDNLHMVNFSNAMBV4UUKFE35JVW7LUSWQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.267773125 (522h9m29.13885616s) [noise=true , address=145.79.227.38:4977   , piece_id=PQJ7M237FOSK7ZDNVKWJEA5FDIWCUPKEDSQM2HRCINFJS4CZHVSQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:30.966667995 (522h9m30.83775103s) [upload=458069]: Next returned nil segment
Dec 27 09:38:30.966678986 (522h9m30.837762021s) [upload=458069]: Waiting for error group managing segments...
Dec 27 09:38:31.255214631 (522h9m31.126297676s) [noise=true , address=145.79.227.46:4601   , piece_id=AAN4BKWIUOOI6F4KJJDYVZ33UFB7HTUNY3L2YWNT5APDPJLZNY4Q, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.263070841 (522h9m31.134153886s) [noise=true , address=145.79.227.29:4943   , piece_id=OCZB6RSLV3VRZJCNH2XTS7QRPB7Y7GWIAVWSW7LL6P2H57SVFTUQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.268523734 (522h9m31.139606769s) [noise=true , address=145.79.227.7:4295    , piece_id=A2YMVUHMSQQNFANVSATKDOJXY2ZOI7KTCCMVVTCLNW6C2Y7CP6WQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.270630546 (522h9m31.141713591s) [noise=true , address=145.79.227.38:4977   , piece_id=PQJ7M237FOSK7ZDNVKWJEA5FDIWCUPKEDSQM2HRCINFJS4CZHVSQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.270780140 (522h9m31.141863175s) [noise=true , address=145.79.227.28:4727   , piece_id=KCODK42JTX4GNR2KZKIHFQ2HD4TTNTBAE2UOIHMC4SCCQL4O2AIQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.286361520 (522h9m31.157444555s) [noise=true , address=145.79.227.37:4567   , piece_id=ZKQ7F32A6F23MBBLR6TATQSIE2YN5E2GVPDLXNCJKJ7LEB27TVUA, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.290523356 (522h9m31.161606411s) [noise=true , address=145.79.227.25:4618   , piece_id=MW2DI5XCWKAK24AUWVVLVPCJXQGG3C6BV47DIUUAZVVJYSVJUAHA, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.292656557 (522h9m31.163739592s) [noise=true , address=145.79.227.39:4015   , piece_id=Z5W6PU6YLBZDFZT6EAWEPP6ZMDLZISBOXRXR6AO7FZCXHBMAOXLA, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.316972079 (522h9m31.188055114s) [noise=true , address=145.79.227.3:4150    , piece_id=P5KXLW74W4FQKI3D5YLBBDNLHMVNFSNAMBV4UUKFE35JVW7LUSWQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.338743536 (522h9m31.209826581s) [noise=true , address=145.79.227.49:4339   , piece_id=WIMRJNYLPIKBXUL7UWAFVAFHWKDX524VIX72QB4HTNWCCCTTFM5Q, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.563259495 (522h9m31.43434254s) [noise=true , address=77.90.31.18:4805     , piece_id=KCFRWQJTVJ5TCUTKYTZOSRZ3HJ3HHZJMN6Q4XWOLAFQHIQQU5HEQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.702721286 (522h9m31.573804331s) [noise=true , address=45.91.116.13:4153    , piece_id=JT4P2BNQFGWVBQGLXSS75KAP32CMDH6GAMJBWYHTBSPSXRFWABAQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:32.234628366 (522h9m32.105711401s) [noise=true , address=77.90.31.15:4697     , piece_id=DX6TFOHKFRW37ES7ZSLV2QIXGSWDJ6TNRFRMG6MAH4LMSOO6PCNQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:32.234658203 (522h9m32.105741238s) [seg_pos={0 0}, upload=458069]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:38:32.234861050 (522h9m32.105944095s) [noise=true , address=45.91.116.4:4937     , piece_id=O26FQGZXDLFTXAE3VN4XNPIGUXU5HKIHP4DX3BLQMB7ACEZMY5RQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 27 09:38:32.234867893 (522h9m32.105950938s) [noise=true , address=45.91.116.4:4937     , piece_id=O26FQGZXDLFTXAE3VN4XNPIGUXU5HKIHP4DX3BLQMB7ACEZMY5RQ, seg_pos={0 0}, upload=458069]: Long tail cancellation due to optimal threshold
Dec 27 09:38:32.235131504 (522h9m32.106214549s) [noise=true , address=45.91.116.10:4081    , piece_id=CR7MTAY6HK3K3ET7QOUYJGSS766Z4OWWUX4PZRHIA3TDAEYH3BIA, seg_pos={0 0}, upload=458069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 27 09:38:32.235141434 (522h9m32.106224469s) [noise=true , address=45.91.116.10:4081    , piece_id=CR7MTAY6HK3K3ET7QOUYJGSS766Z4OWWUX4PZRHIA3TDAEYH3BIA, seg_pos={0 0}, upload=458069]: Long tail cancellation due to optimal threshold
Dec 27 09:38:32.235172623 (522h9m32.106255668s) [noise=true , address=145.79.227.30:4762   , piece_id=VYNQR6MIQIO7IOIOG7PZ2QDHJZZ3PA67AANQXMFBTUUSVOXZZAHQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 09:38:32.235188413 (522h9m32.106271448s) [noise=true , address=145.79.227.30:4762   , piece_id=VYNQR6MIQIO7IOIOG7PZ2QDHJZZ3PA67AANQXMFBTUUSVOXZZAHQ, seg_pos={0 0}, upload=458069]: Long tail cancellation due to optimal threshold
Dec 27 09:38:32.235211938 (522h9m32.106294973s) [seg_pos={0 0}, upload=458069]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:38:32.235322398 (522h9m32.106405443s) [upload=458069]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:38:32.235328700 (522h9m32.106411745s) [upload=458069]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:38:32.528245948 (522h9m32.399328993s) [upload=458069]: Done uploading segments...
Dec 27 09:38:32.528269413 (522h9m32.399352448s) [upload=458069]: Upload finished. err: <nil>
Duration: 6.244519588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.......................X........X.............X.............]
duration: 
38.321212603s
trace: 
https://jaeger.dev.storj.io/trace/71cf25021a4ce946
FAIL (2025-12-27 09:41:02 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2.44819623s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578287911 (522h0m12.449370956s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.578792672 (522h0m12.449875718s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579232119 (522h0m12.450315164s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.579696443 (522h0m12.450779479s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580165196 (522h0m12.451248231s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.580629831 (522h0m12.451712866s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.581852699 (522h0m12.452935744s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.583570440 (522h0m12.454653485s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.584096662 (522h0m12.455179697s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.585109911 (522h0m12.456192956s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.589508798 (522h0m12.460591833s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.590018358 (522h0m12.461101393s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.594544546 (522h0m12.465627591s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:12.596711674 (522h0m12.467794709s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Uploading piece...
Dec 27 09:29:13.920855085 (522h0m13.79193813s) [upload=457930]: Next returned nil segment
Dec 27 09:29:13.920867618 (522h0m13.791950653s) [upload=457930]: Waiting for error group managing segments...
Dec 27 09:29:14.312694120 (522h0m14.183777155s) [noise=true , address=64.49.13.170:4724    , piece_id=AN7MT75FNVUBUQPHOT55AXX3B3KKQEHMKOYIBWNOYGX6KC334VYA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.450082065 (522h0m14.3211651s) [noise=true , address=145.79.227.46:4621   , piece_id=QUJ34P7XUHUESUQUZXRGOFMZ6ZMFC7476DQXKBQ5MJSPF3GNVOZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.457539176 (522h0m14.328622211s) [noise=true , address=145.79.227.23:4548   , piece_id=R6KYDBI7GZFE7PE346KEXSQVGQWOIKDMN7NMN7O3XREUC4ASWI4Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.489586541 (522h0m14.360669576s) [noise=true , address=145.79.227.43:4161   , piece_id=UUIPD246QUCIFCHPRSOEOMU6SE7BYYOGWG7XIS5L6XK6KT7M4ZEQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.498421374 (522h0m14.369504419s) [noise=true , address=145.79.227.33:4894   , piece_id=EHXMIJ7N37FPEOCRSNA4DYCTB7XMB5JR4WABJYLBY53XJPMQ552Q, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.677109595 (522h0m14.54819264s) [noise=true , address=77.90.31.21:4906     , piece_id=47AHXEMQN34KUSBHDP5IM33TPJRY4YGVZMVNBKKKHYC7YV3GOWCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.738116145 (522h0m14.60919919s) [noise=true , address=77.90.31.12:4610     , piece_id=XNFO7XNSTMUBQZ2NXONUP5MMCD4QQYJ2AGXBD7IDCFHSVWCJZMLQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.746701643 (522h0m14.617784678s) [noise=true , address=77.90.31.13:4659     , piece_id=WCW6K2NFKYAA5QQYGFCEW6VPEYR4FAQY5VFA227BRTSASJGIVXFA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:14.797227135 (522h0m14.66831018s) [noise=true , address=77.90.31.11:4579     , piece_id=S3A3FDZBVQ4CDYN5Z5KME6NZ7CZFG3BQKHYR2TRLZLVDHP32GSCA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.067445438 (522h0m14.938528483s) [noise=true , address=77.90.31.19:4850     , piece_id=B5KWCAM3BYZWHB6OST62TSY6JWRPD7PW2AZCRBTTAEKKR55PW5WA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.385254021 (522h0m15.256337056s) [noise=true , address=145.79.227.30:4782   , piece_id=UBDSNSZALLKYPWOJYVTSWIAUFWSMXUFXOLDIBX5KBXGG7XCLVGPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.419727486 (522h0m15.290810521s) [noise=true , address=145.79.227.44:4209   , piece_id=MEEFIKDJKJNXRU4L6MBKAJSULB23DKGM3KQDZGCLCEOBYOD3UEPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.431565969 (522h0m15.302649004s) [noise=true , address=145.79.227.34:4928   , piece_id=KFQ4POASKASJVFBCOR5FX5LJRDL7G4H7LQOULIWFIB5V42J56MQQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457050717 (522h0m15.328133762s) [noise=true , address=145.79.227.40:4053   , piece_id=PDSZEULEQ24GKL52E5X6WIIIGJSBRWV6HRV7XUPCZD3S3PW565RQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: <nil>
Dec 27 09:29:15.457070015 (522h0m15.32815305s) [seg_pos={0 0}, upload=457930]: Segment reached optimal threshold of 14 pieces.
Dec 27 09:29:15.457253995 (522h0m15.32833703s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457262120 (522h0m15.328345155s) [noise=true , address=193.149.164.143:4701 , piece_id=24JJZMFVUH7WTI4RFCP5GZ2F4AFHONZ2NFZAMC3J5LILJO5Y2QPQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457304841 (522h0m15.328387876s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu): protocol: expected piece hash; context canceled; EOF
Dec 27 09:29:15.457312496 (522h0m15.328395531s) [noise=true , address=77.90.31.14:4687     , piece_id=HM7OVJYDFJSOR5OPT2VZZP3QGDMWULSFZ6NNMOPS2KQ3GFILF7ZA, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457368021 (522h0m15.328451057s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS): protocol: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->45.91.116.5:4329: use of closed network connection
Dec 27 09:29:15.457373131 (522h0m15.328456166s) [noise=true , address=45.91.116.5:4329     , piece_id=GGXSOIJ2SSCMZ2L6ASKM72AQOANQRS44EHEOFOYTPLKCMJRTH6FQ, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457473142 (522h0m15.328556187s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48186->64.49.13.169:4705: use of closed network connection
Dec 27 09:29:15.457478892 (522h0m15.328561927s) [noise=true , address=64.49.13.169:4705    , piece_id=RDOB4WDFJ4LU3EFOYPDP4ZDBDLQS3VIT3YMOPK6FRQKZ46BCOK4A, seg_pos={0 0}, upload=457930]: Long tail cancellation due to optimal threshold
Dec 27 09:29:15.457577420 (522h0m15.328660465s) [seg_pos={0 0}, upload=457930]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 09:29:15.457642344 (522h0m15.328725379s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:29:15.457647203 (522h0m15.328730248s) [upload=457930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:29:15.753273248 (522h0m15.624356293s) [upload=457930]: Done uploading segments...
Dec 27 09:29:15.753291964 (522h0m15.624374999s) [upload=457930]: Upload finished. err: <nil>
Duration: 38.32124237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [X.X...X........X.XX...X.X..XXX.X.X.X...X.X.X.X.X.X.X.XXXX...]
duration: 
5.873103902s
trace: 
https://jaeger.dev.storj.io/trace/1906a6556cd44aef
FAIL (2025-12-27 09:40:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: JT4P2BNQFGWVBQGLXSS75KAP32CMDH6GAMJBWYHTBSPSXRFWABAQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: DX6TFOHKFRW37ES7ZSLV2QIXGSWDJ6TNRFRMG6MAH4LMSOO6PCNQ) context deadline exceeded
======LOGS======
Dec 27 09:38:29.005502783 (522h9m28.876585838s) [upload=458069]: Starting upload
Dec 27 09:38:29.005557136 (522h9m28.876640191s) [upload=458069]: Done starting upload
Dec 27 09:38:29.005690470 (522h9m28.876773515s) [upload=458069]: Uploading segments...
Dec 27 09:38:29.005855304 (522h9m28.876938359s) [upload=458069]: Got next segment. Inline: false
Dec 27 09:38:29.005893106 (522h9m28.876976151s) [upload=458069]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 09:38:29.005898687 (522h9m28.876981722s) [upload=458069]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 09:38:29.252809692 (522h9m29.123892727s) [seg_pos={0 0}, upload=458069]: Begin upload segment...
Dec 27 09:38:29.252869115 (522h9m29.12395215s) [seg_pos={0 0}, upload=458069]: Done begin upload segment.
Dec 27 09:38:29.252910173 (522h9m29.123993218s) [noise=true , address=145.79.227.49:4339   , piece_id=WIMRJNYLPIKBXUL7UWAFVAFHWKDX524VIX72QB4HTNWCCCTTFM5Q, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.252957102 (522h9m29.124040147s) [noise=true , address=77.90.31.15:4697     , piece_id=DX6TFOHKFRW37ES7ZSLV2QIXGSWDJ6TNRFRMG6MAH4LMSOO6PCNQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.253628521 (522h9m29.124711566s) [noise=true , address=45.91.116.10:4081    , piece_id=CR7MTAY6HK3K3ET7QOUYJGSS766Z4OWWUX4PZRHIA3TDAEYH3BIA, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.254253771 (522h9m29.125336816s) [noise=true , address=77.90.31.18:4805     , piece_id=KCFRWQJTVJ5TCUTKYTZOSRZ3HJ3HHZJMN6Q4XWOLAFQHIQQU5HEQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.254723125 (522h9m29.125806171s) [noise=true , address=145.79.227.37:4567   , piece_id=ZKQ7F32A6F23MBBLR6TATQSIE2YN5E2GVPDLXNCJKJ7LEB27TVUA, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.255160728 (522h9m29.126243763s) [noise=true , address=45.91.116.4:4937     , piece_id=O26FQGZXDLFTXAE3VN4XNPIGUXU5HKIHP4DX3BLQMB7ACEZMY5RQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.255607519 (522h9m29.126690564s) [noise=true , address=145.79.227.29:4943   , piece_id=OCZB6RSLV3VRZJCNH2XTS7QRPB7Y7GWIAVWSW7LL6P2H57SVFTUQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.256099706 (522h9m29.127182751s) [noise=true , address=145.79.227.28:4727   , piece_id=KCODK42JTX4GNR2KZKIHFQ2HD4TTNTBAE2UOIHMC4SCCQL4O2AIQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.257350708 (522h9m29.128433743s) [noise=true , address=145.79.227.30:4762   , piece_id=VYNQR6MIQIO7IOIOG7PZ2QDHJZZ3PA67AANQXMFBTUUSVOXZZAHQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.257748887 (522h9m29.128831922s) [noise=true , address=145.79.227.25:4618   , piece_id=MW2DI5XCWKAK24AUWVVLVPCJXQGG3C6BV47DIUUAZVVJYSVJUAHA, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.258982876 (522h9m29.130065911s) [noise=true , address=145.79.227.39:4015   , piece_id=Z5W6PU6YLBZDFZT6EAWEPP6ZMDLZISBOXRXR6AO7FZCXHBMAOXLA, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.260452795 (522h9m29.13153583s) [noise=true , address=45.91.116.13:4153    , piece_id=JT4P2BNQFGWVBQGLXSS75KAP32CMDH6GAMJBWYHTBSPSXRFWABAQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.262086436 (522h9m29.133169481s) [noise=true , address=145.79.227.46:4601   , piece_id=AAN4BKWIUOOI6F4KJJDYVZ33UFB7HTUNY3L2YWNT5APDPJLZNY4Q, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.263541056 (522h9m29.134624091s) [noise=true , address=145.79.227.7:4295    , piece_id=A2YMVUHMSQQNFANVSATKDOJXY2ZOI7KTCCMVVTCLNW6C2Y7CP6WQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.264445949 (522h9m29.135528994s) [noise=true , address=145.79.227.3:4150    , piece_id=P5KXLW74W4FQKI3D5YLBBDNLHMVNFSNAMBV4UUKFE35JVW7LUSWQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:29.267773125 (522h9m29.13885616s) [noise=true , address=145.79.227.38:4977   , piece_id=PQJ7M237FOSK7ZDNVKWJEA5FDIWCUPKEDSQM2HRCINFJS4CZHVSQ, seg_pos={0 0}, upload=458069]: Uploading piece...
Dec 27 09:38:30.966667995 (522h9m30.83775103s) [upload=458069]: Next returned nil segment
Dec 27 09:38:30.966678986 (522h9m30.837762021s) [upload=458069]: Waiting for error group managing segments...
Dec 27 09:38:31.255214631 (522h9m31.126297676s) [noise=true , address=145.79.227.46:4601   , piece_id=AAN4BKWIUOOI6F4KJJDYVZ33UFB7HTUNY3L2YWNT5APDPJLZNY4Q, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.263070841 (522h9m31.134153886s) [noise=true , address=145.79.227.29:4943   , piece_id=OCZB6RSLV3VRZJCNH2XTS7QRPB7Y7GWIAVWSW7LL6P2H57SVFTUQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.268523734 (522h9m31.139606769s) [noise=true , address=145.79.227.7:4295    , piece_id=A2YMVUHMSQQNFANVSATKDOJXY2ZOI7KTCCMVVTCLNW6C2Y7CP6WQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.270630546 (522h9m31.141713591s) [noise=true , address=145.79.227.38:4977   , piece_id=PQJ7M237FOSK7ZDNVKWJEA5FDIWCUPKEDSQM2HRCINFJS4CZHVSQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.270780140 (522h9m31.141863175s) [noise=true , address=145.79.227.28:4727   , piece_id=KCODK42JTX4GNR2KZKIHFQ2HD4TTNTBAE2UOIHMC4SCCQL4O2AIQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.286361520 (522h9m31.157444555s) [noise=true , address=145.79.227.37:4567   , piece_id=ZKQ7F32A6F23MBBLR6TATQSIE2YN5E2GVPDLXNCJKJ7LEB27TVUA, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.290523356 (522h9m31.161606411s) [noise=true , address=145.79.227.25:4618   , piece_id=MW2DI5XCWKAK24AUWVVLVPCJXQGG3C6BV47DIUUAZVVJYSVJUAHA, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.292656557 (522h9m31.163739592s) [noise=true , address=145.79.227.39:4015   , piece_id=Z5W6PU6YLBZDFZT6EAWEPP6ZMDLZISBOXRXR6AO7FZCXHBMAOXLA, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.316972079 (522h9m31.188055114s) [noise=true , address=145.79.227.3:4150    , piece_id=P5KXLW74W4FQKI3D5YLBBDNLHMVNFSNAMBV4UUKFE35JVW7LUSWQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.338743536 (522h9m31.209826581s) [noise=true , address=145.79.227.49:4339   , piece_id=WIMRJNYLPIKBXUL7UWAFVAFHWKDX524VIX72QB4HTNWCCCTTFM5Q, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.563259495 (522h9m31.43434254s) [noise=true , address=77.90.31.18:4805     , piece_id=KCFRWQJTVJ5TCUTKYTZOSRZ3HJ3HHZJMN6Q4XWOLAFQHIQQU5HEQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:31.702721286 (522h9m31.573804331s) [noise=true , address=45.91.116.13:4153    , piece_id=JT4P2BNQFGWVBQGLXSS75KAP32CMDH6GAMJBWYHTBSPSXRFWABAQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:32.234628366 (522h9m32.105711401s) [noise=true , address=77.90.31.15:4697     , piece_id=DX6TFOHKFRW37ES7ZSLV2QIXGSWDJ6TNRFRMG6MAH4LMSOO6PCNQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: <nil>
Dec 27 09:38:32.234658203 (522h9m32.105741238s) [seg_pos={0 0}, upload=458069]: Segment reached optimal threshold of 13 pieces.
Dec 27 09:38:32.234861050 (522h9m32.105944095s) [noise=true , address=45.91.116.4:4937     , piece_id=O26FQGZXDLFTXAE3VN4XNPIGUXU5HKIHP4DX3BLQMB7ACEZMY5RQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 27 09:38:32.234867893 (522h9m32.105950938s) [noise=true , address=45.91.116.4:4937     , piece_id=O26FQGZXDLFTXAE3VN4XNPIGUXU5HKIHP4DX3BLQMB7ACEZMY5RQ, seg_pos={0 0}, upload=458069]: Long tail cancellation due to optimal threshold
Dec 27 09:38:32.235131504 (522h9m32.106214549s) [noise=true , address=45.91.116.10:4081    , piece_id=CR7MTAY6HK3K3ET7QOUYJGSS766Z4OWWUX4PZRHIA3TDAEYH3BIA, seg_pos={0 0}, upload=458069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 27 09:38:32.235141434 (522h9m32.106224469s) [noise=true , address=45.91.116.10:4081    , piece_id=CR7MTAY6HK3K3ET7QOUYJGSS766Z4OWWUX4PZRHIA3TDAEYH3BIA, seg_pos={0 0}, upload=458069]: Long tail cancellation due to optimal threshold
Dec 27 09:38:32.235172623 (522h9m32.106255668s) [noise=true , address=145.79.227.30:4762   , piece_id=VYNQR6MIQIO7IOIOG7PZ2QDHJZZ3PA67AANQXMFBTUUSVOXZZAHQ, seg_pos={0 0}, upload=458069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 09:38:32.235188413 (522h9m32.106271448s) [noise=true , address=145.79.227.30:4762   , piece_id=VYNQR6MIQIO7IOIOG7PZ2QDHJZZ3PA67AANQXMFBTUUSVOXZZAHQ, seg_pos={0 0}, upload=458069]: Long tail cancellation due to optimal threshold
Dec 27 09:38:32.235211938 (522h9m32.106294973s) [seg_pos={0 0}, upload=458069]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 09:38:32.235322398 (522h9m32.106405443s) [upload=458069]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 09:38:32.235328700 (522h9m32.106411745s) [upload=458069]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 09:38:32.528245948 (522h9m32.399328993s) [upload=458069]: Done uploading segments...
Dec 27 09:38:32.528269413 (522h9m32.399352448s) [upload=458069]: Upload finished. err: <nil>
Duration: 5.873132055s core uplink us-select-1